May 29 10:12:56.988929 ip-10-0-132-181 systemd[1]: Starting Kubernetes Kubelet...
May 29 10:12:57.434430 ip-10-0-132-181 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 10:12:57.434430 ip-10-0-132-181 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 29 10:12:57.434430 ip-10-0-132-181 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 10:12:57.434430 ip-10-0-132-181 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 29 10:12:57.434430 ip-10-0-132-181 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 10:12:57.437347 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.437261    2562 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 29 10:12:57.443188 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443173    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:12:57.443188 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443188    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443192    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443195    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443198    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443202    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443204    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443207    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443212    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443215    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443218    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443220    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443223    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443226    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443228    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443231    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443233    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443236    2562 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443238    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443241    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:12:57.443252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443243    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443246    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443248    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443251    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443253    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443256    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443260    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443264    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443267    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443270    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443273    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443275    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443278    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443280    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443283    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443285    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443288    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443290    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443292    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:12:57.443735 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443295    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443297    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443302    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443305    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443308    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443312    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443315    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443317    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443320    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443322    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443324    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443327    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443329    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443331    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443334    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443337    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443340    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443342    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443346    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443348    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:12:57.444182 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443350    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443353    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443355    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443357    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443360    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443362    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443365    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443367    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443370    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443372    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443374    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443377    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443379    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443382    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443384    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443387    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443390    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443392    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443395    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443397    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:12:57.444703 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443399    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443402    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443405    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443407    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443410    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443413    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443415    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443801    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443806    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443809    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443812    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443815    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443817    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443820    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443822    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443825    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443828    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443831    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443833    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443836    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:12:57.445179 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443838    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443841    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443843    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443846    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443848    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443852    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443854    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443857    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443860    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443862    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443865    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443867    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443870    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443873    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443877    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443880    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443882    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443887    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443890    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:12:57.445693 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443892    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443895    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443898    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443901    2562 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443903    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443906    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443909    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443911    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443913    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443916    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443918    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443921    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443923    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443926    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443928    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443930    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443933    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443935    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443938    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443941    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:12:57.446195 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443943    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443946    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443949    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443952    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443954    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443957    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443959    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443962    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443964    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443971    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443974    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443976    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443979    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443981    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443983    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443986    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443989    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443992    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443995    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:12:57.446699 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443997    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.443999    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444002    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444004    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444007    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444009    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444011    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444014    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444017    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444019    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444021    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444024    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444026    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444029    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.444031    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445401    2562 flags.go:64] FLAG: --address="0.0.0.0"
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445410    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445415    2562 flags.go:64] FLAG: --anonymous-auth="true"
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445420    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445424    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445427    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 29 10:12:57.447155 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445431    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445436    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445439    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445442    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445446    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445449    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445452    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445455    2562 flags.go:64] FLAG: --cgroup-root=""
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445457    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445460    2562 flags.go:64] FLAG: --client-ca-file=""
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445463    2562 flags.go:64] FLAG: --cloud-config=""
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445466    2562 flags.go:64] FLAG: --cloud-provider="external"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445469    2562 flags.go:64] FLAG: --cluster-dns="[]"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445473    2562 flags.go:64] FLAG: --cluster-domain=""
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445476    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445479    2562 flags.go:64] FLAG: --config-dir=""
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445481    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445485    2562 flags.go:64] FLAG: --container-log-max-files="5"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445489    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445491    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445494    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445498    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445501    2562 flags.go:64] FLAG: --contention-profiling="false"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445504    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 29 10:12:57.447692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445507    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445510    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445513    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445522    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445525    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445528    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445531    2562 flags.go:64] FLAG: --enable-load-reader="false"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445534    2562 flags.go:64] FLAG: --enable-server="true"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445537    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445541    2562 flags.go:64] FLAG: --event-burst="100"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445544    2562 flags.go:64] FLAG: --event-qps="50"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445547    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445550    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445553    2562 flags.go:64] FLAG: --eviction-hard=""
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445557    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445559    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445562    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445565    2562 flags.go:64] FLAG: --eviction-soft=""
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445568    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445571    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445575    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445578    2562 flags.go:64] FLAG: --experimental-mounter-path=""
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445580    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445583    2562 flags.go:64] FLAG: --fail-swap-on="true"
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445586    2562 flags.go:64] FLAG: --feature-gates=""
May 29 10:12:57.448281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445590    2562 flags.go:64] FLAG: --file-check-frequency="20s"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445593    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445596    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445599    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445602    2562 flags.go:64] FLAG: --healthz-port="10248"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445611    2562 flags.go:64] FLAG: --help="false"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445614    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-132-181.ec2.internal"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445617    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445621    2562 flags.go:64] FLAG: --http-check-frequency="20s"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445624    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445627    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445642    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445646    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445649    2562 flags.go:64] FLAG: --image-service-endpoint=""
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445651    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445654    2562 flags.go:64] FLAG: --kube-api-burst="100"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445657    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445660    2562 flags.go:64] FLAG: --kube-api-qps="50"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445663    2562 flags.go:64] FLAG: --kube-reserved=""
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445666    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445668    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445671    2562 flags.go:64] FLAG: --kubelet-cgroups=""
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445675    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445677    2562 flags.go:64] FLAG: --lock-file=""
May 29 10:12:57.448888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445680    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445683    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445686    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445691    2562 flags.go:64] FLAG: --log-json-split-stream="false"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445694    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445697    2562 flags.go:64] FLAG: --log-text-split-stream="false"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445700    2562 flags.go:64] FLAG: --logging-format="text"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445703    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445706    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445709    2562 flags.go:64] FLAG: --manifest-url=""
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445712    2562 flags.go:64] FLAG: --manifest-url-header=""
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445716    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445719    2562 flags.go:64] FLAG: --max-open-files="1000000"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445724    2562 flags.go:64] FLAG: --max-pods="110"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445727    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445730    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445733    2562 flags.go:64] FLAG: --memory-manager-policy="None"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445735    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445738    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445741    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445745    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445751    2562 flags.go:64] FLAG: --node-status-max-images="50"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445754    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445757    2562 flags.go:64] FLAG: --oom-score-adj="-999"
May 29 10:12:57.449466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445760    2562 flags.go:64] FLAG: --pod-cidr=""
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445763    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445768    2562 flags.go:64] FLAG: --pod-manifest-path=""
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445771    2562 flags.go:64] FLAG: --pod-max-pids="-1"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445775    2562 flags.go:64] FLAG: --pods-per-core="0"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445778    2562 flags.go:64] FLAG: --port="10250"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445780    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445783    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06e685c749b67ec61"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445786    2562 flags.go:64] FLAG: --qos-reserved=""
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445789    2562 flags.go:64] FLAG: --read-only-port="10255"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445792    2562 flags.go:64] FLAG: --register-node="true"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445795    2562 flags.go:64] FLAG: --register-schedulable="true"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445797    2562 flags.go:64] FLAG: --register-with-taints=""
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445801    2562 flags.go:64] FLAG: --registry-burst="10"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445804    2562 flags.go:64] FLAG: --registry-qps="5"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445807    2562 flags.go:64] FLAG: --reserved-cpus=""
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445809    2562 flags.go:64] FLAG: --reserved-memory=""
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445813    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445816    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445819    2562 flags.go:64] FLAG: --rotate-certificates="false"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445822    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445824    2562 flags.go:64] FLAG: --runonce="false"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445827    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445830    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445833    2562 flags.go:64] FLAG: --seccomp-default="false"
May 29 10:12:57.450046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445836    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445839    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445842    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445845    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445848    2562 flags.go:64] FLAG: --storage-driver-password="root"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445852    2562 flags.go:64] FLAG: --storage-driver-secure="false"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445855    2562 flags.go:64] FLAG: --storage-driver-table="stats"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445858    2562 flags.go:64] FLAG: --storage-driver-user="root"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445861    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445864    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445867    2562 flags.go:64] FLAG: --system-cgroups=""
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445869    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445875    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445877    2562 flags.go:64] FLAG: --tls-cert-file=""
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445880    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445885    2562 flags.go:64] FLAG: --tls-min-version=""
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445887    2562 flags.go:64] FLAG: --tls-private-key-file=""
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445890    2562 flags.go:64] FLAG: --topology-manager-policy="none"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445893    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445896    2562 flags.go:64] FLAG: --topology-manager-scope="container"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445899    2562 flags.go:64] FLAG: --v="2"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445903    2562 flags.go:64] FLAG: --version="false"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445907    2562 flags.go:64] FLAG: --vmodule=""
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445911    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.445914    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 29 10:12:57.450647 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446012    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446015    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446018    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446021    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446023    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446026    2562 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446029    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446032    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446034    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446036    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446039    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446041    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446046    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446048    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446051    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446054    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446056    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446059    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446061    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446063    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:12:57.451237 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446066    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446068    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446070    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446073    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446077    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446080    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446083    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446086    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446089    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446092    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446094    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446098    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446100    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446103    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446105    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446108    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446110    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446112    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446115    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:12:57.451866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446117    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446120    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446122    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446124    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446127    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446131    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446135    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446138    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446141    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446143    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446146    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446148    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446151    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446153    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446157    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446159    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446162    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446165    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446167    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:12:57.452406 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446169    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446172    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446174    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446177    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446179    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446182    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446184    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446187    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446189    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446192    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446194    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446197    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446199    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446202    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446204    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446207    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446210    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446212    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446217    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446219    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:12:57.453044 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446222    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:12:57.453837 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446232    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:12:57.453837 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446236    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:12:57.453837 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446238    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:12:57.453837 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446241    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:12:57.453837 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446244    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:12:57.453837 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446246    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:12:57.453837 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.446250    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:12:57.453837 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.447007    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 10:12:57.454676 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.454659    2562 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 29 10:12:57.454715 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.454676    2562 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 29 10:12:57.454745 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454725    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:12:57.454745 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454730    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:12:57.454745 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454734    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:12:57.454745 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454737    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:12:57.454745 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454739    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:12:57.454745 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454742    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:12:57.454745 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454745    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:12:57.454745 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454748    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454751    2562 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454753    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454756    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454758    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454761    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454764    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454766    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454769    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454772    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454775    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454777    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454780    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454782    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454785    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454787    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454790    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454792    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454795    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454797    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:12:57.454947 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454800    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454803    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454805    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454808    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454810    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454812    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454815    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454817    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454819    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454822    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454824    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454827    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454829    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454831    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454835    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454837    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454840    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454842    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454845    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454847    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:12:57.455434 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454851    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454854    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454857    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454859    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454862    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454864    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454867    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454869    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454872    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454875    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454878    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454880    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454883    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454885    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454888    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454890    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454892    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454895    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454897    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:12:57.455929 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454900    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454902    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454904    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454907    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454909    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454912    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454914    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454917    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454920    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454922    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454925    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454928    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454930    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454932    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454935    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454937    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454941    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454945    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454948    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:12:57.456383 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.454951    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.454956    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455051    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455056    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455058    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455061    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455064    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455067    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455069    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455072    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455074    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455077    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455080    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455082    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455085    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455087    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:12:57.456866 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455090    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455092    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455094    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455097    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455101    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455105    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455107    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455110    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455113    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455115    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455118    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455120    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455122    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455125    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455127    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455130    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455132    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455134    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455137    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:12:57.457252 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455139    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455142    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455144    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455147    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455149    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455151    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455154    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455156    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455158    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455161    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455163    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455165    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455168    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455170    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455173    2562 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455175    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455179    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455182    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455185    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455188    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:12:57.457722 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455191    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455194    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455198    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455201    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455203    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455206    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455208    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455211    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455213    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455216    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455218    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455220    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455223    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455225    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455227    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455230    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455232    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455234    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455237    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455239    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:12:57.458205 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455241    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455243    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455246    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455248    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455250    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455253    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455255    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455258    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455260    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455263    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455265    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455268    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:57.455270    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.455274    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 10:12:57.458826 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.457047    2562 server.go:962] "Client rotation is on, will bootstrap in background"
May 29 10:12:57.459421 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.459407    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 29 10:12:57.460207 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.460195    2562 server.go:1019] "Starting client certificate rotation"
May 29 10:12:57.460313 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.460299    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 29 10:12:57.460361 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.460331    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 29 10:12:57.486061 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.486039    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 29 10:12:57.490526 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.490504    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 29 10:12:57.516492 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.516470    2562 log.go:25] "Validated CRI v1 runtime API"
May 29 10:12:57.522242 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.522158    2562 log.go:25] "Validated CRI v1 image API"
May 29 10:12:57.523932 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.523911    2562 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 29 10:12:57.530742 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.530723    2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 29 10:12:57.532811 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.532792    2562 fs.go:135] Filesystem UUIDs: map[1f633768-d70c-4fd1-b853-d60a7a651a21:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9ba90348-3037-4186-8691-9fd398c76879:/dev/nvme0n1p4]
May 29 10:12:57.532883 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.532811    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 29 10:12:57.537111 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.537012    2562 manager.go:217] Machine: {Timestamp:2026-05-29 10:12:57.536482571 +0000 UTC m=+0.433042199 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499998 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec287383caea5b1299d778e4e21da4a3 SystemUUID:ec287383-caea-5b12-99d7-78e4e21da4a3 BootID:54aad76d-769f-42bc-a963-3a268d68c88e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:67:f3:0e:1f:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:67:f3:0e:1f:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:d1:60:40:37:44 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 29 10:12:57.537765 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.537754    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 29 10:12:57.537850 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.537838    2562 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 29 10:12:57.538863 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.538838    2562 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 29 10:12:57.539001 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.538864    2562 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-181.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 29 10:12:57.539043 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.539011    2562 topology_manager.go:138] "Creating topology manager with none policy"
May 29 10:12:57.539043 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.539019    2562 container_manager_linux.go:306] "Creating device plugin manager"
May 29 10:12:57.539043 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.539031    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 29 10:12:57.539615 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.539605    2562 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 29 10:12:57.540902 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.540891    2562 state_mem.go:36] "Initialized new in-memory state store"
May 29 10:12:57.541002 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.540993    2562 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 29 10:12:57.543560 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.543550    2562 kubelet.go:491] "Attempting to sync node with API server"
May 29 10:12:57.543602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.543565    2562 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 29 10:12:57.543602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.543576    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 29 10:12:57.543602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.543584    2562 kubelet.go:397] "Adding apiserver pod source"
May 29 10:12:57.543602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.543593    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 29 10:12:57.544840 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.544826    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 29 10:12:57.544840 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.544844    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 29 10:12:57.548024 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.547999    2562 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 29 10:12:57.549605 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.549592    2562 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 29 10:12:57.550309 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550297    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550314    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550320    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550326    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550334    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550340    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550346    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550352    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550359    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550364    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 29 10:12:57.550374 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550373    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 29 10:12:57.550827 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.550657    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 29 10:12:57.551767 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.551754    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 29 10:12:57.551767 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.551765    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 29 10:12:57.551855 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.551771    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjcxz"
May 29 10:12:57.554902 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.554888    2562 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 29 10:12:57.554982 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.554922    2562 server.go:1295] "Started kubelet"
May 29 10:12:57.555033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.554985    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 29 10:12:57.555068 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.554992    2562 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 29 10:12:57.555097 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.555067    2562 server_v1.go:47] "podresources" method="list" useActivePods=true
May 29 10:12:57.555740 ip-10-0-132-181 systemd[1]: Started Kubernetes Kubelet.
May 29 10:12:57.558692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.558669    2562 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 29 10:12:57.558787 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.558741    2562 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-181.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 29 10:12:57.558903 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.558879    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 29 10:12:57.559264 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.559248    2562 server.go:317] "Adding debug handlers to kubelet server"
May 29 10:12:57.559574 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.559550    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-181.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 29 10:12:57.560623 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.560287    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjcxz"
May 29 10:12:57.564018 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.564001    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 29 10:12:57.564018 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.564014    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 29 10:12:57.564148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.564058    2562 volume_manager.go:295] "The desired_state_of_world populator starts"
May 29 10:12:57.564148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.564066    2562 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 29 10:12:57.564488 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.564467    2562 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 29 10:12:57.564820 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.564796    2562 reconstruct.go:97] "Volume reconstruction finished"
May 29 10:12:57.564820 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.564813    2562 reconciler.go:26] "Reconciler: start to sync state"
May 29 10:12:57.565009 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.563859    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-181.ec2.internal.18b401a8e0be738c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-181.ec2.internal,UID:ip-10-0-132-181.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-181.ec2.internal,},FirstTimestamp:2026-05-29 10:12:57.554899852 +0000 UTC m=+0.451459462,LastTimestamp:2026-05-29 10:12:57.554899852 +0000 UTC m=+0.451459462,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-181.ec2.internal,}"
May 29 10:12:57.566438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.566092    2562 factory.go:55] Registering systemd factory
May 29 10:12:57.566438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.566143    2562 factory.go:223] Registration of the systemd container factory successfully
May 29 10:12:57.566438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.566363    2562 factory.go:153] Registering CRI-O factory
May 29 10:12:57.566438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.566376    2562 factory.go:223] Registration of the crio container factory successfully
May 29 10:12:57.566438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.566431    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 29 10:12:57.566842 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.566510    2562 factory.go:103] Registering Raw factory
May 29 10:12:57.566842 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.566522    2562 manager.go:1196] Started watching for new ooms in manager
May 29 10:12:57.566920 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.566887    2562 manager.go:319] Starting recovery of all containers
May 29 10:12:57.568408 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.566358    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:57.575600 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.575576    2562 manager.go:324] Recovery completed
May 29 10:12:57.577703 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.577654    2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
May 29 10:12:57.578331 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.578309    2562 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:12:57.581235 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.581217    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:12:57.581591 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.581576    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-181.ec2.internal\" not found" node="ip-10-0-132-181.ec2.internal"
May 29 10:12:57.583138 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.583123    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:12:57.583202 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.583153    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:12:57.583202 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.583167    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientPID"
May 29 10:12:57.583664 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.583650    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 29 10:12:57.583736 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.583661    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 29 10:12:57.583736 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.583686    2562 state_mem.go:36] "Initialized new in-memory state store"
May 29 10:12:57.586574 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.586562    2562 policy_none.go:49] "None policy: Start"
May 29 10:12:57.586626 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.586577    2562 memory_manager.go:186] "Starting memorymanager" policy="None"
May 29 10:12:57.586626 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.586588    2562 state_mem.go:35] "Initializing new in-memory state store"
May 29 10:12:57.624301 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.624281    2562 manager.go:341] "Starting Device Plugin manager"
May 29 10:12:57.643183 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.624312    2562 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 29 10:12:57.643183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.624323    2562 server.go:85] "Starting device plugin registration server"
May 29 10:12:57.643183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.624588    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
May 29 10:12:57.643183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.624601    2562 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 29 10:12:57.643183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.624839    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 29 10:12:57.643183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.624942    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 29 10:12:57.643183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.624956    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 29 10:12:57.643183 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.625276    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 29 10:12:57.643183 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.625313    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:57.697623 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.697561    2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 29 10:12:57.698921 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.698902    2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 29 10:12:57.699007 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.698935    2562 status_manager.go:230] "Starting to sync pod status with apiserver"
May 29 10:12:57.699007 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.698958    2562 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 29 10:12:57.699007 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.698966    2562 kubelet.go:2451] "Starting kubelet main sync loop"
May 29 10:12:57.699177 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.699012    2562 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 29 10:12:57.705609 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.705590    2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:12:57.725169 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.725153    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:12:57.725982 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.725965    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:12:57.726054 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.726000    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:12:57.726054 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.726011    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientPID"
May 29 10:12:57.726054 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.726033    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-181.ec2.internal"
May 29 10:12:57.734583 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.734568    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-181.ec2.internal"
May 29 10:12:57.734624 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.734591    2562 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-181.ec2.internal\": node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:57.751719 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.751699    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:57.799100 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.799050    2562 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal"]
May 29 10:12:57.799184 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.799140    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:12:57.800785 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.800763    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:12:57.800868 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.800802    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:12:57.800868 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.800815    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientPID"
May 29 10:12:57.801956 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.801944    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:12:57.802120 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.802108    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.802170 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.802135    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:12:57.802595 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.802580    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:12:57.802701 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.802610    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:12:57.802701 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.802625    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientPID"
May 29 10:12:57.802701 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.802651    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:12:57.802701 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.802673    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:12:57.802701 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.802685    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientPID"
May 29 10:12:57.803772 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.803757    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.803847 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.803784    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:12:57.804470 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.804456    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:12:57.804528 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.804479    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:12:57.804528 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.804488    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeHasSufficientPID"
May 29 10:12:57.828176 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.828152    2562 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-181.ec2.internal\" not found" node="ip-10-0-132-181.ec2.internal"
May 29 10:12:57.832507 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.832492    2562 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-181.ec2.internal\" not found" node="ip-10-0-132-181.ec2.internal"
May 29 10:12:57.851890 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.851870    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:57.867016 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.866982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/528fb44b8ad6a2bd21c186b431b5c03e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal\" (UID: \"528fb44b8ad6a2bd21c186b431b5c03e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.867119 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.867024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/523418bbe89940c3ca4a0795cf595797-config\") pod \"kube-apiserver-proxy-ip-10-0-132-181.ec2.internal\" (UID: \"523418bbe89940c3ca4a0795cf595797\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.867119 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.867051    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/528fb44b8ad6a2bd21c186b431b5c03e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal\" (UID: \"528fb44b8ad6a2bd21c186b431b5c03e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.953042 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:57.952968    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:57.967305 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.967285    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/523418bbe89940c3ca4a0795cf595797-config\") pod \"kube-apiserver-proxy-ip-10-0-132-181.ec2.internal\" (UID: \"523418bbe89940c3ca4a0795cf595797\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.967371 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.967309    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/528fb44b8ad6a2bd21c186b431b5c03e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal\" (UID: \"528fb44b8ad6a2bd21c186b431b5c03e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.967371 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.967327    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/528fb44b8ad6a2bd21c186b431b5c03e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal\" (UID: \"528fb44b8ad6a2bd21c186b431b5c03e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.967438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.967372    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/528fb44b8ad6a2bd21c186b431b5c03e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal\" (UID: \"528fb44b8ad6a2bd21c186b431b5c03e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.967438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.967391    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/523418bbe89940c3ca4a0795cf595797-config\") pod \"kube-apiserver-proxy-ip-10-0-132-181.ec2.internal\" (UID: \"523418bbe89940c3ca4a0795cf595797\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal"
May 29 10:12:57.967438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:57.967402    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/528fb44b8ad6a2bd21c186b431b5c03e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal\" (UID: \"528fb44b8ad6a2bd21c186b431b5c03e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"
May 29 10:12:58.053782 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.053738    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:58.131215 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.131191    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"
May 29 10:12:58.135009 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.134990    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal"
May 29 10:12:58.154462 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.154434    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:58.255027 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.254936    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:58.355438 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.355413    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:58.456051 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.456015    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-181.ec2.internal\" not found"
May 29 10:12:58.460165 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.460146    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 29 10:12:58.460287 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.460271    2562 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 29 10:12:58.460348 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.460312    2562 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 29 10:12:58.467815 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.467791    2562 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:12:58.544513 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.544434    2562 apiserver.go:52] "Watching apiserver"
May 29 10:12:58.554347 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.554321    2562 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 29 10:12:58.555382 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.555358    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-5j64w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd","openshift-cluster-node-tuning-operator/tuned-dzdll","openshift-dns/node-resolver-nd22r","openshift-image-registry/node-ca-cxhx6","openshift-network-diagnostics/network-check-target-2j2z5","openshift-ovn-kubernetes/ovnkube-node-2kcfl","openshift-multus/multus-additional-cni-plugins-d5sf6","openshift-multus/multus-v282r","openshift-multus/network-metrics-daemon-wk82g","openshift-network-operator/iptables-alerter-2dcgs"]
May 29 10:12:58.557746 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.557730    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5j64w"
May 29 10:12:58.559969 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.559927    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 29 10:12:58.560072 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.560053    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qqz6m\""
May 29 10:12:58.560508 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.560461    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 29 10:12:58.560903 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.560886    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.562262 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.562218    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-28 10:07:57 +0000 UTC" deadline="2027-12-04 23:04:04.911258991 +0000 UTC"
May 29 10:12:58.562335 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.562264    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13308h51m6.349001603s"
May 29 10:12:58.562478 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.562461    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 29 10:12:58.562559 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.562544    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 29 10:12:58.562697 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.562604    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-st865\""
May 29 10:12:58.562697 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.562627    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 29 10:12:58.562915 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.562897    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.564145 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.564130    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 29 10:12:58.564401 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.564387    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 29 10:12:58.564490 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.564403    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v4pzr\""
May 29 10:12:58.564677 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.564660    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 29 10:12:58.564813 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.564798    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.565523 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.565165    2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"
May 29 10:12:58.566865 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.566828    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 29 10:12:58.567130 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.567115    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9tptk\""
May 29 10:12:58.567214 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.567189    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 29 10:12:58.568005 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.567495    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.569334 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.569320    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 29 10:12:58.569478 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.569457    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tx4lv\""
May 29 10:12:58.569602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.569489    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 29 10:12:58.569739 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.569724    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 29 10:12:58.569998 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.569981    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:12:58.570107 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.570084    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:12:58.570265 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhthz\" (UniqueName: \"kubernetes.io/projected/8aeb7763-facc-47b0-9a47-e654923ca778-kube-api-access-vhthz\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.570329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-modprobe-d\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.570329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdwvh\" (UniqueName: \"kubernetes.io/projected/ff455eab-532e-4f70-bb48-e56d679f294e-kube-api-access-sdwvh\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cwj\" (UniqueName: \"kubernetes.io/projected/27b9a97c-fe5a-4e4c-9979-e69acd4e390b-kube-api-access-q6cwj\") pod \"node-resolver-nd22r\" (UID: \"27b9a97c-fe5a-4e4c-9979-e69acd4e390b\") " pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.570519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570336    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-sys\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-registration-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.570519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570417    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27b9a97c-fe5a-4e4c-9979-e69acd4e390b-hosts-file\") pod \"node-resolver-nd22r\" (UID: \"27b9a97c-fe5a-4e4c-9979-e69acd4e390b\") " pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.570519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-sys-fs\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.570519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-lib-modules\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570495    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17381d52-b51e-4d06-86d0-d814cdef85c5-agent-certs\") pod \"konnectivity-agent-5j64w\" (UID: \"17381d52-b51e-4d06-86d0-d814cdef85c5\") " pod="kube-system/konnectivity-agent-5j64w"
May 29 10:12:58.570519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570516    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17381d52-b51e-4d06-86d0-d814cdef85c5-konnectivity-ca\") pod \"konnectivity-agent-5j64w\" (UID: \"17381d52-b51e-4d06-86d0-d814cdef85c5\") " pod="kube-system/konnectivity-agent-5j64w"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-sysconfig\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-var-lib-kubelet\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570561    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-host\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff455eab-532e-4f70-bb48-e56d679f294e-etc-tuned\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff455eab-532e-4f70-bb48-e56d679f294e-tmp\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b9a97c-fe5a-4e4c-9979-e69acd4e390b-tmp-dir\") pod \"node-resolver-nd22r\" (UID: \"27b9a97c-fe5a-4e4c-9979-e69acd4e390b\") " pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570692    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-device-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-sysctl-d\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-systemd\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-etc-selinux\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-kubernetes\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.570823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570811    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-sysctl-conf\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.571214 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-run\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.571214 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.570907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-socket-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.572032 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.572015    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.574225 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.573970    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 29 10:12:58.574225 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.574052    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kvzvp\""
May 29 10:12:58.574225 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.574092    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 29 10:12:58.574225 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.574106    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 29 10:12:58.574225 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.574053    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 29 10:12:58.574491 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.574363    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 29 10:12:58.574581 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.574567    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.575228 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.575212    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 29 10:12:58.576188 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.576172    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 29 10:12:58.576188 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.576188    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 29 10:12:58.576313 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.576259    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xwmmr\""
May 29 10:12:58.576313 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.576272    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 29 10:12:58.576457 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.576443    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 29 10:12:58.576708 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.576625    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 29 10:12:58.577685 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.577666    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 29 10:12:58.577763 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.577724    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v282r"
May 29 10:12:58.577763 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.577756    2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal"
May 29 10:12:58.578170 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.578154    2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 29 10:12:58.579224 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.579212    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-clnv4\""
May 29 10:12:58.579330 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.579318    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 29 10:12:58.579975 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.579956    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:12:58.580062 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.580043    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:12:58.581996 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.581980    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal"]
May 29 10:12:58.582108 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.582095    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.583663 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.583645    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 29 10:12:58.583825 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.583807    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 29 10:12:58.583907 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.583892    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 29 10:12:58.584059 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.584038    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2wwfx\""
May 29 10:12:58.585598 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.585584    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 29 10:12:58.585693 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.585614    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal"]
May 29 10:12:58.649584 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.649533    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528fb44b8ad6a2bd21c186b431b5c03e.slice/crio-612d6ee18296cc936af34fe98afec18f7efa137553eabb18dbfd2a384e0df08e WatchSource:0}: Error finding container 612d6ee18296cc936af34fe98afec18f7efa137553eabb18dbfd2a384e0df08e: Status 404 returned error can't find the container with id 612d6ee18296cc936af34fe98afec18f7efa137553eabb18dbfd2a384e0df08e
May 29 10:12:58.649828 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.649807    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523418bbe89940c3ca4a0795cf595797.slice/crio-1c4a7326bca16bd1d5021a69b91cf889ea1f080293712d821db9410ca512eed6 WatchSource:0}: Error finding container 1c4a7326bca16bd1d5021a69b91cf889ea1f080293712d821db9410ca512eed6: Status 404 returned error can't find the container with id 1c4a7326bca16bd1d5021a69b91cf889ea1f080293712d821db9410ca512eed6
May 29 10:12:58.654269 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.654252    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 10:12:58.665947 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.665926    2562 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 29 10:12:58.671426 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671405    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-sysctl-conf\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.671510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671435    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-run\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.671510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671454    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-socket-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.671510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671476    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-os-release\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.671510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/603d09a2-9f9f-49c1-b4d2-4d5174cb1f34-iptables-alerter-script\") pod \"iptables-alerter-2dcgs\" (UID: \"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34\") " pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.671686 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671529    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-run\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.671686 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671565    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-sysctl-conf\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.671686 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671568    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/603d09a2-9f9f-49c1-b4d2-4d5174cb1f34-host-slash\") pod \"iptables-alerter-2dcgs\" (UID: \"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34\") " pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.671686 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671615    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b63d125a-0397-4738-92bf-4798f84c67c6-env-overrides\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.671686 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671614    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-socket-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.671686 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671665    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-modprobe-d\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.671923 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671693    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-systemd-units\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.671923 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-cni-netd\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.671923 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-system-cni-dir\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.671923 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671782    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cwj\" (UniqueName: \"kubernetes.io/projected/27b9a97c-fe5a-4e4c-9979-e69acd4e390b-kube-api-access-q6cwj\") pod \"node-resolver-nd22r\" (UID: \"27b9a97c-fe5a-4e4c-9979-e69acd4e390b\") " pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.671923 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671804    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-modprobe-d\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.671923 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671821    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f12b6be-7010-4361-a5e9-e161cb679061-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.671923 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-run-openvswitch\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.672123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-cnibin\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.672123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.671980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27ff9c21-2605-4dd4-a989-e6161b491b8b-cni-binary-copy\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.672123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672009    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-sys\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.672123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672031    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-registration-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.672123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2pbf\" (UniqueName: \"kubernetes.io/projected/27ff9c21-2605-4dd4-a989-e6161b491b8b-kube-api-access-p2pbf\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.672123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672076    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-sys\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.672123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672084    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-registration-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.672123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672094    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27b9a97c-fe5a-4e4c-9979-e69acd4e390b-hosts-file\") pod \"node-resolver-nd22r\" (UID: \"27b9a97c-fe5a-4e4c-9979-e69acd4e390b\") " pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.672123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672121    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-sys-fs\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-system-cni-dir\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672149    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27b9a97c-fe5a-4e4c-9979-e69acd4e390b-hosts-file\") pod \"node-resolver-nd22r\" (UID: \"27b9a97c-fe5a-4e4c-9979-e69acd4e390b\") " pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672170    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722mh\" (UniqueName: \"kubernetes.io/projected/7f12b6be-7010-4361-a5e9-e161cb679061-kube-api-access-722mh\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672220    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-lib-modules\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672200    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-sys-fs\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672257    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-cnibin\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdhq\" (UniqueName: \"kubernetes.io/projected/603d09a2-9f9f-49c1-b4d2-4d5174cb1f34-kube-api-access-6sdhq\") pod \"iptables-alerter-2dcgs\" (UID: \"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34\") " pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5h8x\" (UniqueName: \"kubernetes.io/projected/b63d125a-0397-4738-92bf-4798f84c67c6-kube-api-access-k5h8x\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672325    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-lib-modules\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-hostroot\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.672376 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-run-multus-certs\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672406    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-sysconfig\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672437    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-var-lib-kubelet\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672459    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-var-lib-kubelet\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672464    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b9a97c-fe5a-4e4c-9979-e69acd4e390b-tmp-dir\") pod \"node-resolver-nd22r\" (UID: \"27b9a97c-fe5a-4e4c-9979-e69acd4e390b\") " pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-kubelet\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672440    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-sysconfig\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672515    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-run-netns\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672550    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-var-lib-openvswitch\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672581    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vhthz\" (UniqueName: \"kubernetes.io/projected/8aeb7763-facc-47b0-9a47-e654923ca778-kube-api-access-vhthz\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-etc-openvswitch\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b63d125a-0397-4738-92bf-4798f84c67c6-ovnkube-config\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672715    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672740    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b9a97c-fe5a-4e4c-9979-e69acd4e390b-tmp-dir\") pod \"node-resolver-nd22r\" (UID: \"27b9a97c-fe5a-4e4c-9979-e69acd4e390b\") " pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f12b6be-7010-4361-a5e9-e161cb679061-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672750    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.672881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672783    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sdwvh\" (UniqueName: \"kubernetes.io/projected/ff455eab-532e-4f70-bb48-e56d679f294e-kube-api-access-sdwvh\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e131e33e-9e69-494c-ad8e-865e8f73e4b9-host\") pod \"node-ca-cxhx6\" (UID: \"e131e33e-9e69-494c-ad8e-865e8f73e4b9\") " pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b63d125a-0397-4738-92bf-4798f84c67c6-ovnkube-script-lib\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-run-k8s-cni-cncf-io\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-etc-kubernetes\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqnc6\" (UniqueName: \"kubernetes.io/projected/e131e33e-9e69-494c-ad8e-865e8f73e4b9-kube-api-access-cqnc6\") pod \"node-ca-cxhx6\" (UID: \"e131e33e-9e69-494c-ad8e-865e8f73e4b9\") " pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-run-ovn\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.672997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-log-socket\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b63d125a-0397-4738-92bf-4798f84c67c6-ovn-node-metrics-cert\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-os-release\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-var-lib-cni-multus\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673121    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-var-lib-kubelet\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673142    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-node-log\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-socket-dir-parent\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-conf-dir\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673218    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17381d52-b51e-4d06-86d0-d814cdef85c5-agent-certs\") pod \"konnectivity-agent-5j64w\" (UID: \"17381d52-b51e-4d06-86d0-d814cdef85c5\") " pod="kube-system/konnectivity-agent-5j64w"
May 29 10:12:58.673496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673251    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17381d52-b51e-4d06-86d0-d814cdef85c5-konnectivity-ca\") pod \"konnectivity-agent-5j64w\" (UID: \"17381d52-b51e-4d06-86d0-d814cdef85c5\") " pod="kube-system/konnectivity-agent-5j64w"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e131e33e-9e69-494c-ad8e-865e8f73e4b9-serviceca\") pod \"node-ca-cxhx6\" (UID: \"e131e33e-9e69-494c-ad8e-865e8f73e4b9\") " pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-run-netns\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673333    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-run-systemd\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-cni-bin\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-var-lib-cni-bin\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673409    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-host\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673434    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff455eab-532e-4f70-bb48-e56d679f294e-etc-tuned\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673475    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-host\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673505    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673511    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff455eab-532e-4f70-bb48-e56d679f294e-tmp\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673539    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-device-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673588    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-device-dir\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673644    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-etc-selinux\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nl7\" (UniqueName: \"kubernetes.io/projected/ad608802-fd8d-49b6-945f-b56237be7996-kube-api-access-s7nl7\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673718    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8aeb7763-facc-47b0-9a47-e654923ca778-etc-selinux\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-slash\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673732    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5gzks"
May 29 10:12:58.674171 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673768    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-sysctl-d\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673813    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-systemd\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673828    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17381d52-b51e-4d06-86d0-d814cdef85c5-konnectivity-ca\") pod \"konnectivity-agent-5j64w\" (UID: \"17381d52-b51e-4d06-86d0-d814cdef85c5\") " pod="kube-system/konnectivity-agent-5j64w"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673841    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f12b6be-7010-4361-a5e9-e161cb679061-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673925    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-sysctl-d\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673949    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-systemd\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.673985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-cni-dir\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.674003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-daemon-config\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.674026    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-kubernetes\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.674655 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.674103    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff455eab-532e-4f70-bb48-e56d679f294e-etc-kubernetes\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.676556 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.676537    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff455eab-532e-4f70-bb48-e56d679f294e-etc-tuned\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.676624 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.676574    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17381d52-b51e-4d06-86d0-d814cdef85c5-agent-certs\") pod \"konnectivity-agent-5j64w\" (UID: \"17381d52-b51e-4d06-86d0-d814cdef85c5\") " pod="kube-system/konnectivity-agent-5j64w"
May 29 10:12:58.676624 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.676589    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff455eab-532e-4f70-bb48-e56d679f294e-tmp\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.678618 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.678602    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cwj\" (UniqueName: \"kubernetes.io/projected/27b9a97c-fe5a-4e4c-9979-e69acd4e390b-kube-api-access-q6cwj\") pod \"node-resolver-nd22r\" (UID: \"27b9a97c-fe5a-4e4c-9979-e69acd4e390b\") " pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.682803 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.682786    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5gzks"
May 29 10:12:58.686600 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.686572    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhthz\" (UniqueName: \"kubernetes.io/projected/8aeb7763-facc-47b0-9a47-e654923ca778-kube-api-access-vhthz\") pod \"aws-ebs-csi-driver-node-k9qcd\" (UID: \"8aeb7763-facc-47b0-9a47-e654923ca778\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.686863 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.686845    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdwvh\" (UniqueName: \"kubernetes.io/projected/ff455eab-532e-4f70-bb48-e56d679f294e-kube-api-access-sdwvh\") pod \"tuned-dzdll\" (UID: \"ff455eab-532e-4f70-bb48-e56d679f294e\") " pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.702496 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.702459    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal" event={"ID":"528fb44b8ad6a2bd21c186b431b5c03e","Type":"ContainerStarted","Data":"612d6ee18296cc936af34fe98afec18f7efa137553eabb18dbfd2a384e0df08e"}
May 29 10:12:58.703373 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.703348    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal" event={"ID":"523418bbe89940c3ca4a0795cf595797","Type":"ContainerStarted","Data":"1c4a7326bca16bd1d5021a69b91cf889ea1f080293712d821db9410ca512eed6"}
May 29 10:12:58.774307 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774274    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f12b6be-7010-4361-a5e9-e161cb679061-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.774307 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774311    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-run-openvswitch\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.774514 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774334    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-cnibin\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.774514 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774356    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27ff9c21-2605-4dd4-a989-e6161b491b8b-cni-binary-copy\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.774514 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774401    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-run-openvswitch\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.774514 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774426    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-cnibin\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.774514 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774471    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p2pbf\" (UniqueName: \"kubernetes.io/projected/27ff9c21-2605-4dd4-a989-e6161b491b8b-kube-api-access-p2pbf\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.774514 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774514    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-system-cni-dir\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774534    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-722mh\" (UniqueName: \"kubernetes.io/projected/7f12b6be-7010-4361-a5e9-e161cb679061-kube-api-access-722mh\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774608    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774687    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774685    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-cnibin\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774724    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-cnibin\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774613    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-system-cni-dir\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774743    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdhq\" (UniqueName: \"kubernetes.io/projected/603d09a2-9f9f-49c1-b4d2-4d5174cb1f34-kube-api-access-6sdhq\") pod \"iptables-alerter-2dcgs\" (UID: \"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34\") " pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774763    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k5h8x\" (UniqueName: \"kubernetes.io/projected/b63d125a-0397-4738-92bf-4798f84c67c6-kube-api-access-k5h8x\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774778    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-hostroot\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.774806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774801    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-run-multus-certs\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774833    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-kubelet\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774835    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-hostroot\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774889    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-run-multus-certs\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774898    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-kubelet\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774946    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-run-netns\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774984    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-var-lib-openvswitch\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.774992    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f12b6be-7010-4361-a5e9-e161cb679061-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775002    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27ff9c21-2605-4dd4-a989-e6161b491b8b-cni-binary-copy\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775007    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-etc-openvswitch\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775012    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-run-netns\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775044    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-etc-openvswitch\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775058    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-var-lib-openvswitch\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775077    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775096    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b63d125a-0397-4738-92bf-4798f84c67c6-ovnkube-config\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775116    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f12b6be-7010-4361-a5e9-e161cb679061-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775158    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775163    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e131e33e-9e69-494c-ad8e-865e8f73e4b9-host\") pod \"node-ca-cxhx6\" (UID: \"e131e33e-9e69-494c-ad8e-865e8f73e4b9\") " pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775223    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e131e33e-9e69-494c-ad8e-865e8f73e4b9-host\") pod \"node-ca-cxhx6\" (UID: \"e131e33e-9e69-494c-ad8e-865e8f73e4b9\") " pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775251    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b63d125a-0397-4738-92bf-4798f84c67c6-ovnkube-script-lib\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775297    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-run-k8s-cni-cncf-io\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775325    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-etc-kubernetes\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775375    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cqnc6\" (UniqueName: \"kubernetes.io/projected/e131e33e-9e69-494c-ad8e-865e8f73e4b9-kube-api-access-cqnc6\") pod \"node-ca-cxhx6\" (UID: \"e131e33e-9e69-494c-ad8e-865e8f73e4b9\") " pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775394    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-run-k8s-cni-cncf-io\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775402    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775418    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-etc-kubernetes\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775495    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-run-ovn\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775555    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-run-ovn\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.775574    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775587    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-log-socket\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775614    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b63d125a-0397-4738-92bf-4798f84c67c6-ovn-node-metrics-cert\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775647    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-log-socket\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775661    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-os-release\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775690    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-var-lib-cni-multus\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775721    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-var-lib-kubelet\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.775870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775728    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f12b6be-7010-4361-a5e9-e161cb679061-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775789    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-var-lib-cni-multus\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775779    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-node-log\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775770    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-os-release\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775799    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b63d125a-0397-4738-92bf-4798f84c67c6-ovnkube-config\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775842    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-node-log\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775833    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-socket-dir-parent\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775834    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-var-lib-kubelet\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.775888    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs podName:ad608802-fd8d-49b6-945f-b56237be7996 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:59.275839599 +0000 UTC m=+2.172399216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs") pod "network-metrics-daemon-wk82g" (UID: "ad608802-fd8d-49b6-945f-b56237be7996") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775880    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-socket-dir-parent\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775954    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-conf-dir\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.775974    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-conf-dir\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776029    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e131e33e-9e69-494c-ad8e-865e8f73e4b9-serviceca\") pod \"node-ca-cxhx6\" (UID: \"e131e33e-9e69-494c-ad8e-865e8f73e4b9\") " pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776102    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-run-netns\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776131    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b63d125a-0397-4738-92bf-4798f84c67c6-ovnkube-script-lib\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776130    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-run-systemd\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776188    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-run-systemd\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.776703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776189    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-cni-bin\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776206    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-run-netns\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776225    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-cni-bin\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776258    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-var-lib-cni-bin\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776306    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nl7\" (UniqueName: \"kubernetes.io/projected/ad608802-fd8d-49b6-945f-b56237be7996-kube-api-access-s7nl7\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776334    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-slash\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776359    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776350    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-host-var-lib-cni-bin\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776395    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e131e33e-9e69-494c-ad8e-865e8f73e4b9-serviceca\") pod \"node-ca-cxhx6\" (UID: \"e131e33e-9e69-494c-ad8e-865e8f73e4b9\") " pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776385    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f12b6be-7010-4361-a5e9-e161cb679061-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776471    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776496    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776499    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-cni-dir\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776525    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-daemon-config\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776557    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-os-release\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776585    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-slash\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776664    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-cni-dir\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.777369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776587    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/603d09a2-9f9f-49c1-b4d2-4d5174cb1f34-iptables-alerter-script\") pod \"iptables-alerter-2dcgs\" (UID: \"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34\") " pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776721    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/603d09a2-9f9f-49c1-b4d2-4d5174cb1f34-host-slash\") pod \"iptables-alerter-2dcgs\" (UID: \"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34\") " pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776728    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f12b6be-7010-4361-a5e9-e161cb679061-os-release\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776739    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b63d125a-0397-4738-92bf-4798f84c67c6-env-overrides\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776756    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-systemd-units\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776765    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/603d09a2-9f9f-49c1-b4d2-4d5174cb1f34-host-slash\") pod \"iptables-alerter-2dcgs\" (UID: \"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34\") " pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776774    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-cni-netd\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776790    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-system-cni-dir\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776844    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ff9c21-2605-4dd4-a989-e6161b491b8b-system-cni-dir\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776877    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-systemd-units\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776907    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b63d125a-0397-4738-92bf-4798f84c67c6-host-cni-netd\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.776937    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f12b6be-7010-4361-a5e9-e161cb679061-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.777076    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b63d125a-0397-4738-92bf-4798f84c67c6-env-overrides\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.777157    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/603d09a2-9f9f-49c1-b4d2-4d5174cb1f34-iptables-alerter-script\") pod \"iptables-alerter-2dcgs\" (UID: \"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34\") " pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.777839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.777170    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27ff9c21-2605-4dd4-a989-e6161b491b8b-multus-daemon-config\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.778263 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.778203    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b63d125a-0397-4738-92bf-4798f84c67c6-ovn-node-metrics-cert\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.782837 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.782800    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:12:58.782837 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.782825    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:12:58.782837 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.782837    2562 projected.go:194] Error preparing data for projected volume kube-api-access-9gjkg for pod openshift-network-diagnostics/network-check-target-2j2z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:12:58.783400 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:58.783150    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg podName:eb12e862-d565-46d8-b7da-df8d75835853 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:59.282869336 +0000 UTC m=+2.179428936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9gjkg" (UniqueName: "kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg") pod "network-check-target-2j2z5" (UID: "eb12e862-d565-46d8-b7da-df8d75835853") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:12:58.786684 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.785294    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-722mh\" (UniqueName: \"kubernetes.io/projected/7f12b6be-7010-4361-a5e9-e161cb679061-kube-api-access-722mh\") pod \"multus-additional-cni-plugins-d5sf6\" (UID: \"7f12b6be-7010-4361-a5e9-e161cb679061\") " pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.786684 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.785305    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nl7\" (UniqueName: \"kubernetes.io/projected/ad608802-fd8d-49b6-945f-b56237be7996-kube-api-access-s7nl7\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:12:58.786684 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.786055    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqnc6\" (UniqueName: \"kubernetes.io/projected/e131e33e-9e69-494c-ad8e-865e8f73e4b9-kube-api-access-cqnc6\") pod \"node-ca-cxhx6\" (UID: \"e131e33e-9e69-494c-ad8e-865e8f73e4b9\") " pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.786684 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.786121    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdhq\" (UniqueName: \"kubernetes.io/projected/603d09a2-9f9f-49c1-b4d2-4d5174cb1f34-kube-api-access-6sdhq\") pod \"iptables-alerter-2dcgs\" (UID: \"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34\") " pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.787908 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.787887    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5h8x\" (UniqueName: \"kubernetes.io/projected/b63d125a-0397-4738-92bf-4798f84c67c6-kube-api-access-k5h8x\") pod \"ovnkube-node-2kcfl\" (UID: \"b63d125a-0397-4738-92bf-4798f84c67c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.787983 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.787888    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2pbf\" (UniqueName: \"kubernetes.io/projected/27ff9c21-2605-4dd4-a989-e6161b491b8b-kube-api-access-p2pbf\") pod \"multus-v282r\" (UID: \"27ff9c21-2605-4dd4-a989-e6161b491b8b\") " pod="openshift-multus/multus-v282r"
May 29 10:12:58.888342 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.888314    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5j64w"
May 29 10:12:58.895170 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.895147    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17381d52_b51e_4d06_86d0_d814cdef85c5.slice/crio-f38719131e71ebcd038c24cd55150decb05673c28aa969668cd9dbb89405b585 WatchSource:0}: Error finding container f38719131e71ebcd038c24cd55150decb05673c28aa969668cd9dbb89405b585: Status 404 returned error can't find the container with id f38719131e71ebcd038c24cd55150decb05673c28aa969668cd9dbb89405b585
May 29 10:12:58.899272 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.899255    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd"
May 29 10:12:58.905581 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.905553    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aeb7763_facc_47b0_9a47_e654923ca778.slice/crio-015f07b28ee12d2508474551ab66173e201332388b05fa6ece2e648831b0235f WatchSource:0}: Error finding container 015f07b28ee12d2508474551ab66173e201332388b05fa6ece2e648831b0235f: Status 404 returned error can't find the container with id 015f07b28ee12d2508474551ab66173e201332388b05fa6ece2e648831b0235f
May 29 10:12:58.905662 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.905623    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dzdll"
May 29 10:12:58.911943 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.911918    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff455eab_532e_4f70_bb48_e56d679f294e.slice/crio-4e6a6148e4365450609e30bc9df70fe5d3e1b57e0eb868fcb9f7d8979a74695f WatchSource:0}: Error finding container 4e6a6148e4365450609e30bc9df70fe5d3e1b57e0eb868fcb9f7d8979a74695f: Status 404 returned error can't find the container with id 4e6a6148e4365450609e30bc9df70fe5d3e1b57e0eb868fcb9f7d8979a74695f
May 29 10:12:58.916714 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.916696    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nd22r"
May 29 10:12:58.921273 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.921249    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxhx6"
May 29 10:12:58.923846 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.923809    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b9a97c_fe5a_4e4c_9979_e69acd4e390b.slice/crio-21c9221deb8d8b71bcb70ef2583d5217a3f8c7a9c1e34b3ec80cf26e74bed9c1 WatchSource:0}: Error finding container 21c9221deb8d8b71bcb70ef2583d5217a3f8c7a9c1e34b3ec80cf26e74bed9c1: Status 404 returned error can't find the container with id 21c9221deb8d8b71bcb70ef2583d5217a3f8c7a9c1e34b3ec80cf26e74bed9c1
May 29 10:12:58.927868 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.927839    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode131e33e_9e69_494c_ad8e_865e8f73e4b9.slice/crio-314c36b626350578eedb103bab319b21fcb2feed2316c6f1db7d225f2edd5fa8 WatchSource:0}: Error finding container 314c36b626350578eedb103bab319b21fcb2feed2316c6f1db7d225f2edd5fa8: Status 404 returned error can't find the container with id 314c36b626350578eedb103bab319b21fcb2feed2316c6f1db7d225f2edd5fa8
May 29 10:12:58.955349 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.955306    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:12:58.960991 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.960969    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d5sf6"
May 29 10:12:58.961239 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.961219    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63d125a_0397_4738_92bf_4798f84c67c6.slice/crio-8ab43831d50b51618bd2129920b673be856e75b7fdd57f71c4a09b47c5444680 WatchSource:0}: Error finding container 8ab43831d50b51618bd2129920b673be856e75b7fdd57f71c4a09b47c5444680: Status 404 returned error can't find the container with id 8ab43831d50b51618bd2129920b673be856e75b7fdd57f71c4a09b47c5444680
May 29 10:12:58.967181 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.967162    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v282r"
May 29 10:12:58.967405 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.967375    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f12b6be_7010_4361_a5e9_e161cb679061.slice/crio-2716a49f50e782e59be2568360e760a2622e80aeee1ddc83820561d7f2905c23 WatchSource:0}: Error finding container 2716a49f50e782e59be2568360e760a2622e80aeee1ddc83820561d7f2905c23: Status 404 returned error can't find the container with id 2716a49f50e782e59be2568360e760a2622e80aeee1ddc83820561d7f2905c23
May 29 10:12:58.972257 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.972240    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2dcgs"
May 29 10:12:58.973038 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.973012    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ff9c21_2605_4dd4_a989_e6161b491b8b.slice/crio-b3da18fc4c7a29ac49a7c4c28d4294b139f02cc2d37ad88ebba491505f41a0af WatchSource:0}: Error finding container b3da18fc4c7a29ac49a7c4c28d4294b139f02cc2d37ad88ebba491505f41a0af: Status 404 returned error can't find the container with id b3da18fc4c7a29ac49a7c4c28d4294b139f02cc2d37ad88ebba491505f41a0af
May 29 10:12:58.978215 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:12:58.978181    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603d09a2_9f9f_49c1_b4d2_4d5174cb1f34.slice/crio-41a9655df876236633425e05ce0446f323417153419b37b5779f13243825ef16 WatchSource:0}: Error finding container 41a9655df876236633425e05ce0446f323417153419b37b5779f13243825ef16: Status 404 returned error can't find the container with id 41a9655df876236633425e05ce0446f323417153419b37b5779f13243825ef16
May 29 10:12:58.984265 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:58.984247    2562 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:12:59.268490 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.268411    2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:12:59.280172 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.280143    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:12:59.280346 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:59.280324    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:12:59.280475 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:59.280398    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs podName:ad608802-fd8d-49b6-945f-b56237be7996 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:00.280376551 +0000 UTC m=+3.176936149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs") pod "network-metrics-daemon-wk82g" (UID: "ad608802-fd8d-49b6-945f-b56237be7996") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:12:59.366240 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.365981    2562 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:12:59.381864 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.381235    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:12:59.381864 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:59.381421    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:12:59.381864 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:59.381441    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:12:59.381864 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:59.381459    2562 projected.go:194] Error preparing data for projected volume kube-api-access-9gjkg for pod openshift-network-diagnostics/network-check-target-2j2z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:12:59.381864 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:12:59.381511    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg podName:eb12e862-d565-46d8-b7da-df8d75835853 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:00.381493949 +0000 UTC m=+3.278053557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gjkg" (UniqueName: "kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg") pod "network-check-target-2j2z5" (UID: "eb12e862-d565-46d8-b7da-df8d75835853") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:12:59.686953 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.686915    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 10:07:58 +0000 UTC" deadline="2028-03-05 22:33:15.678915832 +0000 UTC"
May 29 10:12:59.686953 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.686947    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15516h20m15.991971225s"
May 29 10:12:59.735990 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.735923    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxhx6" event={"ID":"e131e33e-9e69-494c-ad8e-865e8f73e4b9","Type":"ContainerStarted","Data":"314c36b626350578eedb103bab319b21fcb2feed2316c6f1db7d225f2edd5fa8"}
May 29 10:12:59.762939 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.762897    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nd22r" event={"ID":"27b9a97c-fe5a-4e4c-9979-e69acd4e390b","Type":"ContainerStarted","Data":"21c9221deb8d8b71bcb70ef2583d5217a3f8c7a9c1e34b3ec80cf26e74bed9c1"}
May 29 10:12:59.775719 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.775681    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5j64w" event={"ID":"17381d52-b51e-4d06-86d0-d814cdef85c5","Type":"ContainerStarted","Data":"f38719131e71ebcd038c24cd55150decb05673c28aa969668cd9dbb89405b585"}
May 29 10:12:59.781285 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.781250    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v282r" event={"ID":"27ff9c21-2605-4dd4-a989-e6161b491b8b","Type":"ContainerStarted","Data":"b3da18fc4c7a29ac49a7c4c28d4294b139f02cc2d37ad88ebba491505f41a0af"}
May 29 10:12:59.786202 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.786170    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5sf6" event={"ID":"7f12b6be-7010-4361-a5e9-e161cb679061","Type":"ContainerStarted","Data":"2716a49f50e782e59be2568360e760a2622e80aeee1ddc83820561d7f2905c23"}
May 29 10:12:59.793008 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.792977    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerStarted","Data":"8ab43831d50b51618bd2129920b673be856e75b7fdd57f71c4a09b47c5444680"}
May 29 10:12:59.800825 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.800794    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dzdll" event={"ID":"ff455eab-532e-4f70-bb48-e56d679f294e","Type":"ContainerStarted","Data":"4e6a6148e4365450609e30bc9df70fe5d3e1b57e0eb868fcb9f7d8979a74695f"}
May 29 10:12:59.809106 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.809074    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd" event={"ID":"8aeb7763-facc-47b0-9a47-e654923ca778","Type":"ContainerStarted","Data":"015f07b28ee12d2508474551ab66173e201332388b05fa6ece2e648831b0235f"}
May 29 10:12:59.823788 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:12:59.823757    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2dcgs" event={"ID":"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34","Type":"ContainerStarted","Data":"41a9655df876236633425e05ce0446f323417153419b37b5779f13243825ef16"}
May 29 10:13:00.293228 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:00.293189    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:00.293411 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:00.293366    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:00.293472 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:00.293432    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs podName:ad608802-fd8d-49b6-945f-b56237be7996 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:02.293412068 +0000 UTC m=+5.189971672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs") pod "network-metrics-daemon-wk82g" (UID: "ad608802-fd8d-49b6-945f-b56237be7996") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:00.395071 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:00.394381    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:00.395071 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:00.394606    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:13:00.395071 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:00.394627    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:13:00.395071 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:00.394655    2562 projected.go:194] Error preparing data for projected volume kube-api-access-9gjkg for pod openshift-network-diagnostics/network-check-target-2j2z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:00.395071 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:00.394716    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg podName:eb12e862-d565-46d8-b7da-df8d75835853 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:02.394698039 +0000 UTC m=+5.291257640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gjkg" (UniqueName: "kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg") pod "network-check-target-2j2z5" (UID: "eb12e862-d565-46d8-b7da-df8d75835853") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:00.688031 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:00.687958    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 10:07:58 +0000 UTC" deadline="2027-10-27 21:30:46.207560302 +0000 UTC"
May 29 10:13:00.688031 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:00.687994    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12395h17m45.51957017s"
May 29 10:13:00.700505 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:00.699863    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:00.700505 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:00.700000    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:00.700505 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:00.700379    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:00.700505 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:00.700464    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:02.309986 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:02.309948    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:02.310416 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:02.310125    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:02.310416 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:02.310213    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs podName:ad608802-fd8d-49b6-945f-b56237be7996 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:06.310193363 +0000 UTC m=+9.206752964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs") pod "network-metrics-daemon-wk82g" (UID: "ad608802-fd8d-49b6-945f-b56237be7996") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:02.413669 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:02.412008    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:02.413669 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:02.412161    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:13:02.413669 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:02.412182    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:13:02.413669 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:02.412191    2562 projected.go:194] Error preparing data for projected volume kube-api-access-9gjkg for pod openshift-network-diagnostics/network-check-target-2j2z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:02.413669 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:02.412250    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg podName:eb12e862-d565-46d8-b7da-df8d75835853 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:06.412229006 +0000 UTC m=+9.308788608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gjkg" (UniqueName: "kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg") pod "network-check-target-2j2z5" (UID: "eb12e862-d565-46d8-b7da-df8d75835853") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:02.700364 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:02.699819    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:02.700364 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:02.699958    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:02.700364 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:02.699818    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:02.700364 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:02.700048    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:04.700579 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:04.699787    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:04.700579 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:04.699976    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:04.700579 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:04.700424    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:04.700579 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:04.700525    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:06.343276 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:06.343236    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:06.343745 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:06.343405    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:06.343745 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:06.343488    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs podName:ad608802-fd8d-49b6-945f-b56237be7996 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:14.343465773 +0000 UTC m=+17.240025377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs") pod "network-metrics-daemon-wk82g" (UID: "ad608802-fd8d-49b6-945f-b56237be7996") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:06.444295 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:06.444255    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:06.444491 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:06.444419    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:13:06.444491 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:06.444437    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:13:06.444491 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:06.444449    2562 projected.go:194] Error preparing data for projected volume kube-api-access-9gjkg for pod openshift-network-diagnostics/network-check-target-2j2z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:06.444671 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:06.444519    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg podName:eb12e862-d565-46d8-b7da-df8d75835853 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:14.444492201 +0000 UTC m=+17.341051815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gjkg" (UniqueName: "kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg") pod "network-check-target-2j2z5" (UID: "eb12e862-d565-46d8-b7da-df8d75835853") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:06.699819 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:06.699779    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:06.699987 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:06.699905    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:06.700320 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:06.700297    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:06.700450 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:06.700398    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:08.699819 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:08.699790    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:08.700256 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:08.699800    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:08.700256 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:08.699897    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:08.700256 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:08.699976    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:10.699306 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:10.699265    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:10.699686 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:10.699384    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:10.699686 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:10.699456    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:10.699686 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:10.699579    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:12.699991 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:12.699953    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:12.699991 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:12.699981    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:12.700477 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:12.700064    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:12.700477 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:12.700161    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:14.405885 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:14.405853    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:14.406361 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:14.406033    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:14.406361 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:14.406113    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs podName:ad608802-fd8d-49b6-945f-b56237be7996 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:30.406091477 +0000 UTC m=+33.302651080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs") pod "network-metrics-daemon-wk82g" (UID: "ad608802-fd8d-49b6-945f-b56237be7996") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:14.506369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:14.506334    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:14.506543 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:14.506474    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:13:14.506543 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:14.506497    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:13:14.506543 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:14.506509    2562 projected.go:194] Error preparing data for projected volume kube-api-access-9gjkg for pod openshift-network-diagnostics/network-check-target-2j2z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:14.506737 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:14.506576    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg podName:eb12e862-d565-46d8-b7da-df8d75835853 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:30.506557889 +0000 UTC m=+33.403117499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gjkg" (UniqueName: "kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg") pod "network-check-target-2j2z5" (UID: "eb12e862-d565-46d8-b7da-df8d75835853") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:14.699546 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:14.699462    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:14.699712 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:14.699615    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:14.699712 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:14.699683    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:14.699835 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:14.699787    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:16.699522 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:16.699482    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:16.699947 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:16.699535    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:16.699947 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:16.699653    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:16.699947 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:16.699780    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:18.700090 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:18.700063    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:18.700447 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:18.700068    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:18.700447 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:18.700157    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:18.700447 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:18.700229    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:18.861609 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:18.861582    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerStarted","Data":"a4eb3e32923d9007538d71d90a889fcb11f957eefa7a62025eb85cd3ae42d55c"}
May 29 10:13:18.862967 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:18.862945    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dzdll" event={"ID":"ff455eab-532e-4f70-bb48-e56d679f294e","Type":"ContainerStarted","Data":"e2ccd5ea17bf5a72c3babf8b747311ddba89b3ea45eb9deb73601a7264c8ca82"}
May 29 10:13:18.864096 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:18.864078    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal" event={"ID":"523418bbe89940c3ca4a0795cf595797","Type":"ContainerStarted","Data":"e1766e9bea032d8108a816d7024eedfddd27526bdb838779b99ca46bbdb18a38"}
May 29 10:13:18.881239 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:18.880973    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dzdll" podStartSLOduration=2.135804124 podStartE2EDuration="21.880955495s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:12:58.913345974 +0000 UTC m=+1.809905570" lastFinishedPulling="2026-05-29 10:13:18.658497326 +0000 UTC m=+21.555056941" observedRunningTime="2026-05-29 10:13:18.880479735 +0000 UTC m=+21.777039353" watchObservedRunningTime="2026-05-29 10:13:18.880955495 +0000 UTC m=+21.777515118"
May 29 10:13:18.893458 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:18.893233    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-181.ec2.internal" podStartSLOduration=20.893218669 podStartE2EDuration="20.893218669s" podCreationTimestamp="2026-05-29 10:12:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:13:18.893119791 +0000 UTC m=+21.789679412" watchObservedRunningTime="2026-05-29 10:13:18.893218669 +0000 UTC m=+21.789778288"
May 29 10:13:19.868774 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.868444    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:13:19.869482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.869040    2562 generic.go:358] "Generic (PLEG): container finished" podID="b63d125a-0397-4738-92bf-4798f84c67c6" containerID="7e8b81250bddeb8dada461c98dbd720de5d6a2bcc815a8f3c5ad7f640f8eb3fd" exitCode=1
May 29 10:13:19.869482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.869095    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerStarted","Data":"80c64bba0c51bbdc8b718bd54202c415ef913933defbf865424cb1644bd1e3cd"}
May 29 10:13:19.869482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.869122    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerStarted","Data":"efb5bae71b5430ffe91ec64cb08fd566da7d4743778e101f532bc8aef21102c8"}
May 29 10:13:19.869482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.869136    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerStarted","Data":"c4c9596fb006b6b89df22e3800acfd8285160b0e52840a4c67f90da532da8150"}
May 29 10:13:19.869482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.869149    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerStarted","Data":"68b0b0829890cd4ce4bf19e79177e3be7a24bf65ad6cf25e12cfec3c23ebf324"}
May 29 10:13:19.869482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.869162    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerDied","Data":"7e8b81250bddeb8dada461c98dbd720de5d6a2bcc815a8f3c5ad7f640f8eb3fd"}
May 29 10:13:19.870264 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.870243    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd" event={"ID":"8aeb7763-facc-47b0-9a47-e654923ca778","Type":"ContainerStarted","Data":"3df4fe9afab4abb426d657ac0752e33677d17e345179cc33cf7936b8b0074676"}
May 29 10:13:19.871363 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.871342    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxhx6" event={"ID":"e131e33e-9e69-494c-ad8e-865e8f73e4b9","Type":"ContainerStarted","Data":"351d2e1519c96f2f6d30990ec6b09aa278f6e2cc98a31c64ddd91d18af77f3db"}
May 29 10:13:19.872468 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.872450    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nd22r" event={"ID":"27b9a97c-fe5a-4e4c-9979-e69acd4e390b","Type":"ContainerStarted","Data":"ef558e2e31cf9f1c3c1bcfee6b3fe7aa72f1bb2fbd6424d9b343d0f87531cab5"}
May 29 10:13:19.873541 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.873522    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5j64w" event={"ID":"17381d52-b51e-4d06-86d0-d814cdef85c5","Type":"ContainerStarted","Data":"8a7631aba5b0700e6527a6eeb9ed52c00e4e27bc34aa967ffcab53dae69a2c60"}
May 29 10:13:19.874708 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.874688    2562 generic.go:358] "Generic (PLEG): container finished" podID="528fb44b8ad6a2bd21c186b431b5c03e" containerID="f71159d4e97fabf5e68f848621ff996f8b03e5787bc37729301a47c50f69fde2" exitCode=0
May 29 10:13:19.874806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.874743    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal" event={"ID":"528fb44b8ad6a2bd21c186b431b5c03e","Type":"ContainerDied","Data":"f71159d4e97fabf5e68f848621ff996f8b03e5787bc37729301a47c50f69fde2"}
May 29 10:13:19.875926 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.875908    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v282r" event={"ID":"27ff9c21-2605-4dd4-a989-e6161b491b8b","Type":"ContainerStarted","Data":"2cfa4d4a55d3175e6e1808c345b746e3a464e84c11ce4008c182e80b93787fdd"}
May 29 10:13:19.877176 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.877154    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f12b6be-7010-4361-a5e9-e161cb679061" containerID="0cb57624bdf7dffa608b82aed9d587acadebfe0e183921ff40af97852999513e" exitCode=0
May 29 10:13:19.877258 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.877229    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5sf6" event={"ID":"7f12b6be-7010-4361-a5e9-e161cb679061","Type":"ContainerDied","Data":"0cb57624bdf7dffa608b82aed9d587acadebfe0e183921ff40af97852999513e"}
May 29 10:13:19.909277 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.909238    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cxhx6" podStartSLOduration=11.615382292 podStartE2EDuration="22.909223612s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:12:58.929515739 +0000 UTC m=+1.826075336" lastFinishedPulling="2026-05-29 10:13:10.223357055 +0000 UTC m=+13.119916656" observedRunningTime="2026-05-29 10:13:19.887400166 +0000 UTC m=+22.783959785" watchObservedRunningTime="2026-05-29 10:13:19.909223612 +0000 UTC m=+22.805783288"
May 29 10:13:19.933671 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.933593    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-v282r" podStartSLOduration=3.057629636 podStartE2EDuration="22.933575056s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:12:58.974991359 +0000 UTC m=+1.871550956" lastFinishedPulling="2026-05-29 10:13:18.850936779 +0000 UTC m=+21.747496376" observedRunningTime="2026-05-29 10:13:19.910561678 +0000 UTC m=+22.807121297" watchObservedRunningTime="2026-05-29 10:13:19.933575056 +0000 UTC m=+22.830134676"
May 29 10:13:19.965772 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.965714    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5j64w" podStartSLOduration=11.639166607 podStartE2EDuration="22.965695483s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:12:58.896836216 +0000 UTC m=+1.793395813" lastFinishedPulling="2026-05-29 10:13:10.223365075 +0000 UTC m=+13.119924689" observedRunningTime="2026-05-29 10:13:19.965285727 +0000 UTC m=+22.861845346" watchObservedRunningTime="2026-05-29 10:13:19.965695483 +0000 UTC m=+22.862255103"
May 29 10:13:19.979420 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:19.979380    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-nd22r" podStartSLOduration=3.262615884 podStartE2EDuration="22.97936731s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:12:58.925278147 +0000 UTC m=+1.821837744" lastFinishedPulling="2026-05-29 10:13:18.642029567 +0000 UTC m=+21.538589170" observedRunningTime="2026-05-29 10:13:19.979073748 +0000 UTC m=+22.875633368" watchObservedRunningTime="2026-05-29 10:13:19.97936731 +0000 UTC m=+22.875926928"
May 29 10:13:20.346182 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.346157    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 29 10:13:20.638907 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.638792    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-29T10:13:20.346175443Z","UUID":"b97a42fe-6407-4edf-9afa-aa32224151fd","Handler":null,"Name":"","Endpoint":""}
May 29 10:13:20.641478 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.641450    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 29 10:13:20.641478 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.641485    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 29 10:13:20.699874 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.699850    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:20.700019 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.699857    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:20.700019 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:20.699933    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:20.700101 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:20.700015    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:20.881943 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.881904    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd" event={"ID":"8aeb7763-facc-47b0-9a47-e654923ca778","Type":"ContainerStarted","Data":"6bc1344d29ba8e9f61474705c7a0b99ba4ed26f49f7c25bab4e131f69d3e49fc"}
May 29 10:13:20.883426 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.883385    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2dcgs" event={"ID":"603d09a2-9f9f-49c1-b4d2-4d5174cb1f34","Type":"ContainerStarted","Data":"58f2beedd558b7796e7b76ab048d9569bb0635e04a29690fac517ddccfcd9f03"}
May 29 10:13:20.886554 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.886533    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal" event={"ID":"528fb44b8ad6a2bd21c186b431b5c03e","Type":"ContainerStarted","Data":"f733c664a22a5b7d103996d77a9630a6618dd3de65f43ad3458bab9c213792c9"}
May 29 10:13:20.898482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.898398    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2dcgs" podStartSLOduration=4.219563613 podStartE2EDuration="23.898383755s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:12:58.979557316 +0000 UTC m=+1.876116916" lastFinishedPulling="2026-05-29 10:13:18.658377451 +0000 UTC m=+21.554937058" observedRunningTime="2026-05-29 10:13:20.89782308 +0000 UTC m=+23.794382702" watchObservedRunningTime="2026-05-29 10:13:20.898383755 +0000 UTC m=+23.794943373"
May 29 10:13:20.912883 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:20.912836    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-181.ec2.internal" podStartSLOduration=22.912824326 podStartE2EDuration="22.912824326s" podCreationTimestamp="2026-05-29 10:12:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:13:20.912221429 +0000 UTC m=+23.808781049" watchObservedRunningTime="2026-05-29 10:13:20.912824326 +0000 UTC m=+23.809383945"
May 29 10:13:21.890538 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:21.890508    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:13:21.891011 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:21.890986    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerStarted","Data":"f082f336917dcd3c9e221c23be05e99cc0cf64d2a3356ea2256e90208b484073"}
May 29 10:13:21.892967 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:21.892937    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd" event={"ID":"8aeb7763-facc-47b0-9a47-e654923ca778","Type":"ContainerStarted","Data":"6feddd131abcc6982dc2816bb7452f468e2f2c7a15033605501e61e2fb90faff"}
May 29 10:13:21.921391 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:21.921329    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9qcd" podStartSLOduration=2.406589516 podStartE2EDuration="24.9213121s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:12:58.907246655 +0000 UTC m=+1.803806251" lastFinishedPulling="2026-05-29 10:13:21.421969223 +0000 UTC m=+24.318528835" observedRunningTime="2026-05-29 10:13:21.921094971 +0000 UTC m=+24.817654590" watchObservedRunningTime="2026-05-29 10:13:21.9213121 +0000 UTC m=+24.817871721"
May 29 10:13:22.700002 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:22.699964    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:22.700183 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:22.700082    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:22.700327 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:22.700301    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:22.700448 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:22.700414    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:23.042555 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:23.042475    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5j64w"
May 29 10:13:23.043339 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:23.043322    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5j64w"
May 29 10:13:23.897084 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:23.897048    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5j64w"
May 29 10:13:23.897590 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:23.897557    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5j64w"
May 29 10:13:24.699288 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.699123    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:24.699854 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.699123    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:24.699854 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:24.699381    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:24.699854 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:24.699523    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:24.899520 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.899451    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f12b6be-7010-4361-a5e9-e161cb679061" containerID="452bce6d64423b8f27e5fcf7a8820aaf41430f6344d0e92296e227dc3b0bc226" exitCode=0
May 29 10:13:24.899653 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.899522    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5sf6" event={"ID":"7f12b6be-7010-4361-a5e9-e161cb679061","Type":"ContainerDied","Data":"452bce6d64423b8f27e5fcf7a8820aaf41430f6344d0e92296e227dc3b0bc226"}
May 29 10:13:24.902552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.902534    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:13:24.903074 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.903028    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerStarted","Data":"ebffbd4122e1dcc8a4bc911f4d52c3dae7b1b6d76233d1a581cb4b6034a2c87f"}
May 29 10:13:24.903272 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.903234    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:13:24.903355 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.903279    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:13:24.903355 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.903294    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:13:24.903433 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.903364    2562 scope.go:117] "RemoveContainer" containerID="7e8b81250bddeb8dada461c98dbd720de5d6a2bcc815a8f3c5ad7f640f8eb3fd"
May 29 10:13:24.919431 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.919411    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:13:24.920503 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:24.920479    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:13:25.907270 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:25.907243    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:13:25.907888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:25.907541    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" event={"ID":"b63d125a-0397-4738-92bf-4798f84c67c6","Type":"ContainerStarted","Data":"5e6afdedf96366539fa5ddd5ace900704cda877bc90081b7affadc800cf25194"}
May 29 10:13:26.699169 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:26.699132    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:26.699169 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:26.699169    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:26.699468 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:26.699262    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:26.699468 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:26.699373    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:26.910681 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:26.910623    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f12b6be-7010-4361-a5e9-e161cb679061" containerID="9dd4170de4220564891dc0dff269f842faa159f03f0742b50bde28a79809fb8e" exitCode=0
May 29 10:13:26.911075 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:26.910713    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5sf6" event={"ID":"7f12b6be-7010-4361-a5e9-e161cb679061","Type":"ContainerDied","Data":"9dd4170de4220564891dc0dff269f842faa159f03f0742b50bde28a79809fb8e"}
May 29 10:13:26.930207 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:26.930169    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl" podStartSLOduration=10.192359862 podStartE2EDuration="29.930156826s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:12:58.964054827 +0000 UTC m=+1.860614425" lastFinishedPulling="2026-05-29 10:13:18.701851776 +0000 UTC m=+21.598411389" observedRunningTime="2026-05-29 10:13:25.936230795 +0000 UTC m=+28.832790415" watchObservedRunningTime="2026-05-29 10:13:26.930156826 +0000 UTC m=+29.826716502"
May 29 10:13:28.700154 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:28.699976    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:28.700537 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:28.699972    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:28.700537 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:28.700270    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:28.700537 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:28.700301    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:28.916249 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:28.916220    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f12b6be-7010-4361-a5e9-e161cb679061" containerID="0d33b0f414cdbcae064aaeabcf51cf6f2ee871064c7b71e7d6dc7515167bd776" exitCode=0
May 29 10:13:28.916401 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:28.916254    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5sf6" event={"ID":"7f12b6be-7010-4361-a5e9-e161cb679061","Type":"ContainerDied","Data":"0d33b0f414cdbcae064aaeabcf51cf6f2ee871064c7b71e7d6dc7515167bd776"}
May 29 10:13:30.425027 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:30.424990    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:30.425504 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:30.425162    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:30.425504 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:30.425225    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs podName:ad608802-fd8d-49b6-945f-b56237be7996 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:02.425206713 +0000 UTC m=+65.321766320 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs") pod "network-metrics-daemon-wk82g" (UID: "ad608802-fd8d-49b6-945f-b56237be7996") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:13:30.525850 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:30.525813    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:30.526034 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:30.525946    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:13:30.526034 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:30.525961    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:13:30.526034 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:30.525971    2562 projected.go:194] Error preparing data for projected volume kube-api-access-9gjkg for pod openshift-network-diagnostics/network-check-target-2j2z5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:30.526034 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:30.526031    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg podName:eb12e862-d565-46d8-b7da-df8d75835853 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:02.526011029 +0000 UTC m=+65.422570648 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gjkg" (UniqueName: "kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg") pod "network-check-target-2j2z5" (UID: "eb12e862-d565-46d8-b7da-df8d75835853") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:13:30.699388 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:30.699277    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:30.699388 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:30.699293    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:30.699608 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:30.699398    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:30.699608 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:30.699518    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:32.700042 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:32.700008    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:32.700042 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:32.700028    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:32.700539 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:32.700136    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:32.700539 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:32.700276    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:34.699929 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:34.699898    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:34.700285 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:34.699907    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:34.700285 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:34.700042    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:34.700285 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:34.700166    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:36.699647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:36.699588    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:36.699647 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:36.699588    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:36.700071 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:36.699743    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:36.700071 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:36.699851    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:38.699156 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:38.699122    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:38.699156 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:38.699138    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:38.699612 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:38.699227    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:38.699612 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:38.699378    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:39.940622 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:39.940591    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f12b6be-7010-4361-a5e9-e161cb679061" containerID="08bddc7eefdefaf8d98e8d114062605dad2393b3d38bbc4ce1782379940ff7c0" exitCode=0
May 29 10:13:39.941005 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:39.940670    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5sf6" event={"ID":"7f12b6be-7010-4361-a5e9-e161cb679061","Type":"ContainerDied","Data":"08bddc7eefdefaf8d98e8d114062605dad2393b3d38bbc4ce1782379940ff7c0"}
May 29 10:13:40.386002 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:40.385972    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2j2z5"]
May 29 10:13:40.386189 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:40.386085    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:40.386189 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:40.386170    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:40.389423 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:40.389391    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wk82g"]
May 29 10:13:40.389547 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:40.389493    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:40.389611 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:40.389585    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:40.945507 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:40.945470    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f12b6be-7010-4361-a5e9-e161cb679061" containerID="dbfe3b65a394a715c1871f4664fb65da1b7056ec74d487eeb5ca58a8e26085f3" exitCode=0
May 29 10:13:40.945953 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:40.945519    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5sf6" event={"ID":"7f12b6be-7010-4361-a5e9-e161cb679061","Type":"ContainerDied","Data":"dbfe3b65a394a715c1871f4664fb65da1b7056ec74d487eeb5ca58a8e26085f3"}
May 29 10:13:41.699400 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:41.699225    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:41.699759 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:41.699264    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:41.699759 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:41.699476    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:41.699759 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:41.699559    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:41.950164 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:41.950083    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5sf6" event={"ID":"7f12b6be-7010-4361-a5e9-e161cb679061","Type":"ContainerStarted","Data":"e7e5ce3d9f52e4d838980b4f7239871e3776ba076649a1d0cdefd38e950f70b8"}
May 29 10:13:41.974523 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:41.974475    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d5sf6" podStartSLOduration=4.399330761 podStartE2EDuration="44.974461879s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:12:58.968808168 +0000 UTC m=+1.865367765" lastFinishedPulling="2026-05-29 10:13:39.543939283 +0000 UTC m=+42.440498883" observedRunningTime="2026-05-29 10:13:41.974320011 +0000 UTC m=+44.870879629" watchObservedRunningTime="2026-05-29 10:13:41.974461879 +0000 UTC m=+44.871021498"
May 29 10:13:43.699723 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:43.699685    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:43.699723 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:43.699720    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:43.700132 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:43.699787    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j2z5" podUID="eb12e862-d565-46d8-b7da-df8d75835853"
May 29 10:13:43.700132 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:43.699907    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wk82g" podUID="ad608802-fd8d-49b6-945f-b56237be7996"
May 29 10:13:44.425287 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.425258    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-181.ec2.internal" event="NodeReady"
May 29 10:13:44.425467 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.425394    2562 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 29 10:13:44.470016 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.469982    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qxnsw"]
May 29 10:13:44.490036 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.490009    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8272x"]
May 29 10:13:44.490200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.490180    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.492078 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.492049    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 29 10:13:44.492389 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.492097    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p8r89\""
May 29 10:13:44.492389 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.492097    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 29 10:13:44.503997 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.503976    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qxnsw"]
May 29 10:13:44.504119 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.504045    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8272x"]
May 29 10:13:44.504164 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.504119    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8272x"
May 29 10:13:44.505915 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.505889    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-65d8n\""
May 29 10:13:44.506083 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.505986    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 29 10:13:44.506083 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.506021    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 29 10:13:44.506083 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.506033    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 29 10:13:44.634195 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.634163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vztj6\" (UniqueName: \"kubernetes.io/projected/19b92d9f-b8be-47e1-b184-0f0769490fd6-kube-api-access-vztj6\") pod \"ingress-canary-8272x\" (UID: \"19b92d9f-b8be-47e1-b184-0f0769490fd6\") " pod="openshift-ingress-canary/ingress-canary-8272x"
May 29 10:13:44.634359 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.634209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/facacd1b-46f4-46af-9447-11f7a0245ab2-metrics-tls\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.634359 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.634241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19b92d9f-b8be-47e1-b184-0f0769490fd6-cert\") pod \"ingress-canary-8272x\" (UID: \"19b92d9f-b8be-47e1-b184-0f0769490fd6\") " pod="openshift-ingress-canary/ingress-canary-8272x"
May 29 10:13:44.634359 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.634298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/facacd1b-46f4-46af-9447-11f7a0245ab2-config-volume\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.634359 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.634320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facacd1b-46f4-46af-9447-11f7a0245ab2-tmp-dir\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.634359 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.634336    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqm4\" (UniqueName: \"kubernetes.io/projected/facacd1b-46f4-46af-9447-11f7a0245ab2-kube-api-access-tqqm4\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.734871 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.734836    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/facacd1b-46f4-46af-9447-11f7a0245ab2-metrics-tls\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.734871 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.734876    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19b92d9f-b8be-47e1-b184-0f0769490fd6-cert\") pod \"ingress-canary-8272x\" (UID: \"19b92d9f-b8be-47e1-b184-0f0769490fd6\") " pod="openshift-ingress-canary/ingress-canary-8272x"
May 29 10:13:44.735510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.734915    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/facacd1b-46f4-46af-9447-11f7a0245ab2-config-volume\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.735510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.734932    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facacd1b-46f4-46af-9447-11f7a0245ab2-tmp-dir\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.735510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.734947    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqm4\" (UniqueName: \"kubernetes.io/projected/facacd1b-46f4-46af-9447-11f7a0245ab2-kube-api-access-tqqm4\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.735510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.735001    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vztj6\" (UniqueName: \"kubernetes.io/projected/19b92d9f-b8be-47e1-b184-0f0769490fd6-kube-api-access-vztj6\") pod \"ingress-canary-8272x\" (UID: \"19b92d9f-b8be-47e1-b184-0f0769490fd6\") " pod="openshift-ingress-canary/ingress-canary-8272x"
May 29 10:13:44.735510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.735328    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facacd1b-46f4-46af-9447-11f7a0245ab2-tmp-dir\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.735732 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.735528    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/facacd1b-46f4-46af-9447-11f7a0245ab2-config-volume\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.739080 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.739046    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/facacd1b-46f4-46af-9447-11f7a0245ab2-metrics-tls\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.739185 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.739168    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19b92d9f-b8be-47e1-b184-0f0769490fd6-cert\") pod \"ingress-canary-8272x\" (UID: \"19b92d9f-b8be-47e1-b184-0f0769490fd6\") " pod="openshift-ingress-canary/ingress-canary-8272x"
May 29 10:13:44.744244 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.744223    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqm4\" (UniqueName: \"kubernetes.io/projected/facacd1b-46f4-46af-9447-11f7a0245ab2-kube-api-access-tqqm4\") pod \"dns-default-qxnsw\" (UID: \"facacd1b-46f4-46af-9447-11f7a0245ab2\") " pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.744350 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.744330    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vztj6\" (UniqueName: \"kubernetes.io/projected/19b92d9f-b8be-47e1-b184-0f0769490fd6-kube-api-access-vztj6\") pod \"ingress-canary-8272x\" (UID: \"19b92d9f-b8be-47e1-b184-0f0769490fd6\") " pod="openshift-ingress-canary/ingress-canary-8272x"
May 29 10:13:44.799336 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.799298    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:44.814148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.814103    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8272x"
May 29 10:13:44.966336 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.966097    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qxnsw"]
May 29 10:13:44.969155 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:44.969131    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8272x"]
May 29 10:13:44.970455 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:44.970426    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacacd1b_46f4_46af_9447_11f7a0245ab2.slice/crio-042dfe051f3eec819320b3fba27c02a991cfb3a866dc5e1490eb63a8a11fa72c WatchSource:0}: Error finding container 042dfe051f3eec819320b3fba27c02a991cfb3a866dc5e1490eb63a8a11fa72c: Status 404 returned error can't find the container with id 042dfe051f3eec819320b3fba27c02a991cfb3a866dc5e1490eb63a8a11fa72c
May 29 10:13:44.972583 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:44.972556    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b92d9f_b8be_47e1_b184_0f0769490fd6.slice/crio-897305f1b58b26afbd5805077431542ecb2ac1e34081e114fb32243806ac5188 WatchSource:0}: Error finding container 897305f1b58b26afbd5805077431542ecb2ac1e34081e114fb32243806ac5188: Status 404 returned error can't find the container with id 897305f1b58b26afbd5805077431542ecb2ac1e34081e114fb32243806ac5188
May 29 10:13:45.141812 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.141779    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ffd4fc95-jtqxf"]
May 29 10:13:45.156436 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.156411    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ffd4fc95-jtqxf"]
May 29 10:13:45.156569 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.156519    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.158741 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.158708    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 29 10:13:45.158870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.158779    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hsxgr\""
May 29 10:13:45.159099 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.159066    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 29 10:13:45.159220 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.159111    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 29 10:13:45.159560 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.159508    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 29 10:13:45.159727 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.159708    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 29 10:13:45.159881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.159791    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 29 10:13:45.160007 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.159990    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 29 10:13:45.165402 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.165383    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 29 10:13:45.338710 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.338672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvm2z\" (UniqueName: \"kubernetes.io/projected/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-kube-api-access-fvm2z\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.338873 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.338766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-service-ca\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.338873 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.338796    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-trusted-ca-bundle\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.338961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.338871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-oauth-config\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.338961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.338898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-config\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.338961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.338918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-oauth-serving-cert\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.338961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.338939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-serving-cert\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.440198 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.440057    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fvm2z\" (UniqueName: \"kubernetes.io/projected/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-kube-api-access-fvm2z\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.440198 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.440144    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-service-ca\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.440198 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.440176    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-trusted-ca-bundle\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.440546 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.440214    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-oauth-config\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.440546 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.440245    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-config\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.440546 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.440270    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-oauth-serving-cert\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.440546 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.440299    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-serving-cert\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.441111 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.441049    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-service-ca\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.441111 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.441051    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-config\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.441274 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.441197    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-trusted-ca-bundle\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.441329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.441274    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-oauth-serving-cert\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.444109 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.444091    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-serving-cert\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.444413 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.444387    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-oauth-config\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.449738 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.449688    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvm2z\" (UniqueName: \"kubernetes.io/projected/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-kube-api-access-fvm2z\") pod \"console-7ffd4fc95-jtqxf\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") " pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.467246 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.467219    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:45.607483 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.607444    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ffd4fc95-jtqxf"]
May 29 10:13:45.611731 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:45.611699    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d23793b_1b40_4b96_bcdd_3a224b3bfefc.slice/crio-9fb9dd7a0599429b7ff7d082ec0baf9b7fcd8c17c0d9987eee2d632d55e2f932 WatchSource:0}: Error finding container 9fb9dd7a0599429b7ff7d082ec0baf9b7fcd8c17c0d9987eee2d632d55e2f932: Status 404 returned error can't find the container with id 9fb9dd7a0599429b7ff7d082ec0baf9b7fcd8c17c0d9987eee2d632d55e2f932
May 29 10:13:45.699734 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.699441    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:13:45.699734 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.699478    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:13:45.702658 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.701968    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 29 10:13:45.702658 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.701989    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 29 10:13:45.702658 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.701968    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cf54m\""
May 29 10:13:45.702658 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.702017    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hmwm9\""
May 29 10:13:45.702658 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.702291    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 29 10:13:45.958918 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.958520    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qxnsw" event={"ID":"facacd1b-46f4-46af-9447-11f7a0245ab2","Type":"ContainerStarted","Data":"042dfe051f3eec819320b3fba27c02a991cfb3a866dc5e1490eb63a8a11fa72c"}
May 29 10:13:45.959822 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.959783    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffd4fc95-jtqxf" event={"ID":"8d23793b-1b40-4b96-bcdd-3a224b3bfefc","Type":"ContainerStarted","Data":"9fb9dd7a0599429b7ff7d082ec0baf9b7fcd8c17c0d9987eee2d632d55e2f932"}
May 29 10:13:45.960833 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:45.960799    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8272x" event={"ID":"19b92d9f-b8be-47e1-b184-0f0769490fd6","Type":"ContainerStarted","Data":"897305f1b58b26afbd5805077431542ecb2ac1e34081e114fb32243806ac5188"}
May 29 10:13:48.261762 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:48.261717    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"]
May 29 10:13:48.308036 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:48.307991    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"]
May 29 10:13:48.308200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:48.308132    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"
May 29 10:13:48.310370 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:48.310344    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h4tdm\""
May 29 10:13:48.310370 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:48.310369    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 29 10:13:48.464690 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:48.464625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dabfe1b7-f676-4956-80e8-fb95d00ea4a9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv\" (UID: \"dabfe1b7-f676-4956-80e8-fb95d00ea4a9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"
May 29 10:13:48.565421 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:48.565342    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dabfe1b7-f676-4956-80e8-fb95d00ea4a9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv\" (UID: \"dabfe1b7-f676-4956-80e8-fb95d00ea4a9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"
May 29 10:13:48.565540 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:48.565494    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 29 10:13:48.565575 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:48.565556    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabfe1b7-f676-4956-80e8-fb95d00ea4a9-tls-certificates podName:dabfe1b7-f676-4956-80e8-fb95d00ea4a9 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:49.065541776 +0000 UTC m=+51.962101372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/dabfe1b7-f676-4956-80e8-fb95d00ea4a9-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv" (UID: "dabfe1b7-f676-4956-80e8-fb95d00ea4a9") : secret "prometheus-operator-admission-webhook-tls" not found
May 29 10:13:49.069892 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:49.069821    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dabfe1b7-f676-4956-80e8-fb95d00ea4a9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv\" (UID: \"dabfe1b7-f676-4956-80e8-fb95d00ea4a9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"
May 29 10:13:49.070009 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:49.069979    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 29 10:13:49.070073 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:49.070060    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabfe1b7-f676-4956-80e8-fb95d00ea4a9-tls-certificates podName:dabfe1b7-f676-4956-80e8-fb95d00ea4a9 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:50.070040541 +0000 UTC m=+52.966600159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/dabfe1b7-f676-4956-80e8-fb95d00ea4a9-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv" (UID: "dabfe1b7-f676-4956-80e8-fb95d00ea4a9") : secret "prometheus-operator-admission-webhook-tls" not found
May 29 10:13:49.970742 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:49.970537    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffd4fc95-jtqxf" event={"ID":"8d23793b-1b40-4b96-bcdd-3a224b3bfefc","Type":"ContainerStarted","Data":"404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e"}
May 29 10:13:49.971918 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:49.971897    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8272x" event={"ID":"19b92d9f-b8be-47e1-b184-0f0769490fd6","Type":"ContainerStarted","Data":"84d59768bf030fd095d13d1db74f58fff20fe70c201af7b49301d3a0bd2cdeeb"}
May 29 10:13:49.973344 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:49.973323    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qxnsw" event={"ID":"facacd1b-46f4-46af-9447-11f7a0245ab2","Type":"ContainerStarted","Data":"ac80451d9d4263c3c9a1f394e396ce683064d12271792392fb804d924248c1e9"}
May 29 10:13:49.973344 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:49.973347    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qxnsw" event={"ID":"facacd1b-46f4-46af-9447-11f7a0245ab2","Type":"ContainerStarted","Data":"fa9e33aa10a30a6bb43d1e9efebf707c409b66b3564ae1a2342aaca94a7df86d"}
May 29 10:13:49.973479 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:49.973441    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qxnsw"
May 29 10:13:49.986592 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:49.986547    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7ffd4fc95-jtqxf" podStartSLOduration=1.564034109 podStartE2EDuration="4.986535185s" podCreationTimestamp="2026-05-29 10:13:45 +0000 UTC" firstStartedPulling="2026-05-29 10:13:45.614245675 +0000 UTC m=+48.510805275" lastFinishedPulling="2026-05-29 10:13:49.036746754 +0000 UTC m=+51.933306351" observedRunningTime="2026-05-29 10:13:49.986356909 +0000 UTC m=+52.882916528" watchObservedRunningTime="2026-05-29 10:13:49.986535185 +0000 UTC m=+52.883094804"
May 29 10:13:50.000450 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:50.000402    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8272x" podStartSLOduration=1.945036932 podStartE2EDuration="6.000390877s" podCreationTimestamp="2026-05-29 10:13:44 +0000 UTC" firstStartedPulling="2026-05-29 10:13:44.974107303 +0000 UTC m=+47.870666900" lastFinishedPulling="2026-05-29 10:13:49.029461248 +0000 UTC m=+51.926020845" observedRunningTime="2026-05-29 10:13:49.999926188 +0000 UTC m=+52.896485807" watchObservedRunningTime="2026-05-29 10:13:50.000390877 +0000 UTC m=+52.896950497"
May 29 10:13:50.019214 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:50.019169    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-qxnsw" podStartSLOduration=1.962118179 podStartE2EDuration="6.019154993s" podCreationTimestamp="2026-05-29 10:13:44 +0000 UTC" firstStartedPulling="2026-05-29 10:13:44.97225062 +0000 UTC m=+47.868810230" lastFinishedPulling="2026-05-29 10:13:49.029287448 +0000 UTC m=+51.925847044" observedRunningTime="2026-05-29 10:13:50.018756246 +0000 UTC m=+52.915315865" watchObservedRunningTime="2026-05-29 10:13:50.019154993 +0000 UTC m=+52.915714612"
May 29 10:13:50.079439 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:50.079404    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dabfe1b7-f676-4956-80e8-fb95d00ea4a9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv\" (UID: \"dabfe1b7-f676-4956-80e8-fb95d00ea4a9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"
May 29 10:13:50.082801 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:50.082772    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dabfe1b7-f676-4956-80e8-fb95d00ea4a9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv\" (UID: \"dabfe1b7-f676-4956-80e8-fb95d00ea4a9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"
May 29 10:13:50.118666 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:50.118614    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"
May 29 10:13:50.243086 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:50.242914    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"]
May 29 10:13:50.255191 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:50.255149    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabfe1b7_f676_4956_80e8_fb95d00ea4a9.slice/crio-87e1e1f7e0d8ec74aa26ec787c776458840cdc07f720e91d56d01720228fc963 WatchSource:0}: Error finding container 87e1e1f7e0d8ec74aa26ec787c776458840cdc07f720e91d56d01720228fc963: Status 404 returned error can't find the container with id 87e1e1f7e0d8ec74aa26ec787c776458840cdc07f720e91d56d01720228fc963
May 29 10:13:50.977060 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:50.977022    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv" event={"ID":"dabfe1b7-f676-4956-80e8-fb95d00ea4a9","Type":"ContainerStarted","Data":"87e1e1f7e0d8ec74aa26ec787c776458840cdc07f720e91d56d01720228fc963"}
May 29 10:13:50.997222 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:50.997189    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7ffd4fc95-jtqxf_8d23793b-1b40-4b96-bcdd-3a224b3bfefc/console/0.log"
May 29 10:13:51.779074 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:51.778998    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qxnsw_facacd1b-46f4-46af-9447-11f7a0245ab2/dns/0.log"
May 29 10:13:51.981275 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:51.981241    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv" event={"ID":"dabfe1b7-f676-4956-80e8-fb95d00ea4a9","Type":"ContainerStarted","Data":"deae2fb1c64d5069a465f3a217a1292892b2008bd01ac7dbd75a48c021c01db4"}
May 29 10:13:51.981275 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:51.981255    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qxnsw_facacd1b-46f4-46af-9447-11f7a0245ab2/kube-rbac-proxy/0.log"
May 29 10:13:51.981781 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:51.981395    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"
May 29 10:13:51.986779 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:51.986753    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv"
May 29 10:13:52.041193 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.041084    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv" podStartSLOduration=2.849455634 podStartE2EDuration="4.041069461s" podCreationTimestamp="2026-05-29 10:13:48 +0000 UTC" firstStartedPulling="2026-05-29 10:13:50.257134855 +0000 UTC m=+53.153694452" lastFinishedPulling="2026-05-29 10:13:51.448748678 +0000 UTC m=+54.345308279" observedRunningTime="2026-05-29 10:13:52.005652694 +0000 UTC m=+54.902212306" watchObservedRunningTime="2026-05-29 10:13:52.041069461 +0000 UTC m=+54.937629079"
May 29 10:13:52.349348 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.349319    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"]
May 29 10:13:52.351387 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.351347    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.360574 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.360544    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"]
May 29 10:13:52.364362 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.364339    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 29 10:13:52.364475 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.364365    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 29 10:13:52.364475 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.364419    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-k2nlk\""
May 29 10:13:52.364475 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.364373    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 29 10:13:52.364838 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.364824    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 29 10:13:52.365070 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.365054    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 29 10:13:52.397106 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.397077    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95552ec2-9eee-4ee6-beee-1464cd7878ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.397261 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.397130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95552ec2-9eee-4ee6-beee-1464cd7878ee-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.397261 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.397165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhqv\" (UniqueName: \"kubernetes.io/projected/95552ec2-9eee-4ee6-beee-1464cd7878ee-kube-api-access-6hhqv\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.397261 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.397182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95552ec2-9eee-4ee6-beee-1464cd7878ee-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.497922 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.497876    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhqv\" (UniqueName: \"kubernetes.io/projected/95552ec2-9eee-4ee6-beee-1464cd7878ee-kube-api-access-6hhqv\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.497922 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.497929    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95552ec2-9eee-4ee6-beee-1464cd7878ee-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.498066 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.497968    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95552ec2-9eee-4ee6-beee-1464cd7878ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.498066 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.498007    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95552ec2-9eee-4ee6-beee-1464cd7878ee-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.498585 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.498560    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95552ec2-9eee-4ee6-beee-1464cd7878ee-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.501529 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.501508    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95552ec2-9eee-4ee6-beee-1464cd7878ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.501656 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.501530    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95552ec2-9eee-4ee6-beee-1464cd7878ee-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.505844 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.505822    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhqv\" (UniqueName: \"kubernetes.io/projected/95552ec2-9eee-4ee6-beee-1464cd7878ee-kube-api-access-6hhqv\") pod \"prometheus-operator-7779bbdd78-d4dwc\" (UID: \"95552ec2-9eee-4ee6-beee-1464cd7878ee\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.661047 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.660958    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"
May 29 10:13:52.782169 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.782138    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc"]
May 29 10:13:52.782766 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.782752    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nd22r_27b9a97c-fe5a-4e4c-9979-e69acd4e390b/dns-node-resolver/0.log"
May 29 10:13:52.785041 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:52.785021    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95552ec2_9eee_4ee6_beee_1464cd7878ee.slice/crio-d1ad64dd0a82da73a31c554a3b403b6c9a158d7a8eed39a8c3a76c6c2a6c93fe WatchSource:0}: Error finding container d1ad64dd0a82da73a31c554a3b403b6c9a158d7a8eed39a8c3a76c6c2a6c93fe: Status 404 returned error can't find the container with id d1ad64dd0a82da73a31c554a3b403b6c9a158d7a8eed39a8c3a76c6c2a6c93fe
May 29 10:13:52.984877 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:52.984794    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc" event={"ID":"95552ec2-9eee-4ee6-beee-1464cd7878ee","Type":"ContainerStarted","Data":"d1ad64dd0a82da73a31c554a3b403b6c9a158d7a8eed39a8c3a76c6c2a6c93fe"}
May 29 10:13:53.978867 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:53.978840    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxhx6_e131e33e-9e69-494c-ad8e-865e8f73e4b9/node-ca/0.log"
May 29 10:13:53.990419 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:53.990385    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc" event={"ID":"95552ec2-9eee-4ee6-beee-1464cd7878ee","Type":"ContainerStarted","Data":"93c748c81c3f1731b044abc8138f7f96633cc7e17277c3a810ffe55fea521a02"}
May 29 10:13:54.578858 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.578830    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8272x_19b92d9f-b8be-47e1-b184-0f0769490fd6/serve-healthcheck-canary/0.log"
May 29 10:13:54.688116 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.688088    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z69ss"]
May 29 10:13:54.690206 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.690189    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.692134 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.692112    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 29 10:13:54.692257 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.692134    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xfg85\""
May 29 10:13:54.692257 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.692111    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 29 10:13:54.692569 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.692552    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 29 10:13:54.692769 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.692747    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 29 10:13:54.698745 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.698722    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z69ss"]
May 29 10:13:54.715414 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.715389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjsg\" (UniqueName: \"kubernetes.io/projected/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-kube-api-access-wvjsg\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.715542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.715446    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.715542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.715492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-data-volume\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.715542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.715518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.715766 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.715562    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-crio-socket\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.816241 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.816193    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-crio-socket\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.816465 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.816260    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjsg\" (UniqueName: \"kubernetes.io/projected/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-kube-api-access-wvjsg\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.816465 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.816296    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.816465 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.816314    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-data-volume\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.816465 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.816330    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.816465 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:54.816431    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 29 10:13:54.816465 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.816451    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-crio-socket\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.816717 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:54.816488    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-insights-runtime-extractor-tls podName:88ed25c8-f538-4c47-82a7-f41c3ec07cf4 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:55.31647077 +0000 UTC m=+58.213030367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z69ss" (UID: "88ed25c8-f538-4c47-82a7-f41c3ec07cf4") : secret "insights-runtime-extractor-tls" not found
May 29 10:13:54.816768 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.816739    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-data-volume\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.816910 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.816893    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.824252 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.824235    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjsg\" (UniqueName: \"kubernetes.io/projected/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-kube-api-access-wvjsg\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:54.995703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:54.995670    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc" event={"ID":"95552ec2-9eee-4ee6-beee-1464cd7878ee","Type":"ContainerStarted","Data":"88c320f0ffb3201612408cc6df4433f93509866513e336adc6066e3da8f0b446"}
May 29 10:13:55.019148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:55.019102    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-d4dwc" podStartSLOduration=1.891378209 podStartE2EDuration="3.019087353s" podCreationTimestamp="2026-05-29 10:13:52 +0000 UTC" firstStartedPulling="2026-05-29 10:13:52.78686269 +0000 UTC m=+55.683422301" lastFinishedPulling="2026-05-29 10:13:53.914571846 +0000 UTC m=+56.811131445" observedRunningTime="2026-05-29 10:13:55.018378145 +0000 UTC m=+57.914937799" watchObservedRunningTime="2026-05-29 10:13:55.019087353 +0000 UTC m=+57.915646966"
May 29 10:13:55.319208 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:55.319128    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:55.319352 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:55.319249    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 29 10:13:55.319352 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:55.319315    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-insights-runtime-extractor-tls podName:88ed25c8-f538-4c47-82a7-f41c3ec07cf4 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:56.319297669 +0000 UTC m=+59.215857266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z69ss" (UID: "88ed25c8-f538-4c47-82a7-f41c3ec07cf4") : secret "insights-runtime-extractor-tls" not found
May 29 10:13:55.467430 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:55.467397    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:55.467430 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:55.467435    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:55.472323 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:55.472301    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:56.003150 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.003123    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:13:56.326165 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.326080    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:56.328565 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.328543    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ed25c8-f538-4c47-82a7-f41c3ec07cf4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z69ss\" (UID: \"88ed25c8-f538-4c47-82a7-f41c3ec07cf4\") " pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:56.499296 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.499254    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z69ss"
May 29 10:13:56.630596 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.630562    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z69ss"]
May 29 10:13:56.633753 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:56.633728    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ed25c8_f538_4c47_82a7_f41c3ec07cf4.slice/crio-0ad1919bd6e44897ffbef5b332768d9f7247109d31363af5c048725c7ba3fa9c WatchSource:0}: Error finding container 0ad1919bd6e44897ffbef5b332768d9f7247109d31363af5c048725c7ba3fa9c: Status 404 returned error can't find the container with id 0ad1919bd6e44897ffbef5b332768d9f7247109d31363af5c048725c7ba3fa9c
May 29 10:13:56.728482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.728448    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"]
May 29 10:13:56.757994 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.757971    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"]
May 29 10:13:56.758127 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.758107    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.760537 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.760515    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 29 10:13:56.760659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.760515    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5r5vj\""
May 29 10:13:56.760659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.760515    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 29 10:13:56.769097 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.769073    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ddr92"]
May 29 10:13:56.790387 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.790354    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"]
May 29 10:13:56.790552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.790533    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.792509 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.792488    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 29 10:13:56.792715 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.792697    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jwqzg\""
May 29 10:13:56.792818 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.792799    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 29 10:13:56.792959 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.792836    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 29 10:13:56.812345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.812319    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"]
May 29 10:13:56.812470 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.812437    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.814843 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.814821    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 29 10:13:56.814965 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.814947    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 29 10:13:56.815035 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.815019    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kjk2p\""
May 29 10:13:56.815088 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.814947    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 29 10:13:56.829542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-accelerators-collector-config\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.829542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78b1c61d-cdb6-4f3d-928d-8fc041990dad-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.829799 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829564    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtx98\" (UniqueName: \"kubernetes.io/projected/78b1c61d-cdb6-4f3d-928d-8fc041990dad-kube-api-access-wtx98\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.829799 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.829799 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-wtmp\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.829799 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829683    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/178970c5-e1e7-4175-9613-37ef79e22aab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.829799 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/178970c5-e1e7-4175-9613-37ef79e22aab-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.829799 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-metrics-client-ca\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.829799 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-root\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.830140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-textfile\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.830140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfm4n\" (UniqueName: \"kubernetes.io/projected/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-kube-api-access-tfm4n\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.830140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78b1c61d-cdb6-4f3d-928d-8fc041990dad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.830140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/178970c5-e1e7-4175-9613-37ef79e22aab-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.830140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829920    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/178970c5-e1e7-4175-9613-37ef79e22aab-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.830140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.829947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n689q\" (UniqueName: \"kubernetes.io/projected/178970c5-e1e7-4175-9613-37ef79e22aab-kube-api-access-n689q\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.830140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.830003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78b1c61d-cdb6-4f3d-928d-8fc041990dad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.830140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.830057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-sys\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.830140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.830128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/178970c5-e1e7-4175-9613-37ef79e22aab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.830448 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.830193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-tls\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.931083 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931052    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-textfile\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.931083 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931079    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tfm4n\" (UniqueName: \"kubernetes.io/projected/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-kube-api-access-tfm4n\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.931298 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931098    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78b1c61d-cdb6-4f3d-928d-8fc041990dad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.931298 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931119    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/178970c5-e1e7-4175-9613-37ef79e22aab-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.931298 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:56.931205    2562 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 29 10:13:56.931298 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:56.931275    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b1c61d-cdb6-4f3d-928d-8fc041990dad-openshift-state-metrics-tls podName:78b1c61d-cdb6-4f3d-928d-8fc041990dad nodeName:}" failed. No retries permitted until 2026-05-29 10:13:57.431259363 +0000 UTC m=+60.327818960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/78b1c61d-cdb6-4f3d-928d-8fc041990dad-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-qs7t7" (UID: "78b1c61d-cdb6-4f3d-928d-8fc041990dad") : secret "openshift-state-metrics-tls" not found
May 29 10:13:56.931499 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931325    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/178970c5-e1e7-4175-9613-37ef79e22aab-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.931499 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931365    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n689q\" (UniqueName: \"kubernetes.io/projected/178970c5-e1e7-4175-9613-37ef79e22aab-kube-api-access-n689q\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.931499 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931399    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78b1c61d-cdb6-4f3d-928d-8fc041990dad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.931499 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931424    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-sys\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.931499 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931425    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/178970c5-e1e7-4175-9613-37ef79e22aab-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.931499 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931486    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-sys\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.931809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931502    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-textfile\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.931809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931529    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/178970c5-e1e7-4175-9613-37ef79e22aab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.931809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931585    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-tls\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.931809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931668    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-accelerators-collector-config\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.931809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931701    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78b1c61d-cdb6-4f3d-928d-8fc041990dad-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.931809 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:56.931707    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 29 10:13:56.931809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931728    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wtx98\" (UniqueName: \"kubernetes.io/projected/78b1c61d-cdb6-4f3d-928d-8fc041990dad-kube-api-access-wtx98\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.931809 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:56.931755    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-tls podName:5e768df9-bc5a-4816-9c9e-c5df5a4d8c35 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:57.431738643 +0000 UTC m=+60.328298247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-tls") pod "node-exporter-ddr92" (UID: "5e768df9-bc5a-4816-9c9e-c5df5a4d8c35") : secret "node-exporter-tls" not found
May 29 10:13:56.931809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931785    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.932394 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931818    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-wtmp\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.932394 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931849    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/178970c5-e1e7-4175-9613-37ef79e22aab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.932394 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931882    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/178970c5-e1e7-4175-9613-37ef79e22aab-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.932394 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931946    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-metrics-client-ca\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.932394 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.931978    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-root\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.932394 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.932058    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-root\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.932394 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.932078    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-wtmp\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.932935 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.932413    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/178970c5-e1e7-4175-9613-37ef79e22aab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.932935 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.932443    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-accelerators-collector-config\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.932935 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.932477    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78b1c61d-cdb6-4f3d-928d-8fc041990dad-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.933096 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.933021    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/178970c5-e1e7-4175-9613-37ef79e22aab-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.933096 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.933043    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-metrics-client-ca\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.933270 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.933251    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kcfl"
May 29 10:13:56.934215 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.934189    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/178970c5-e1e7-4175-9613-37ef79e22aab-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.934309 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.934251    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78b1c61d-cdb6-4f3d-928d-8fc041990dad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.934418 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.934401    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.934418 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.934410    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/178970c5-e1e7-4175-9613-37ef79e22aab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:56.952924 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.952904    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfm4n\" (UniqueName: \"kubernetes.io/projected/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-kube-api-access-tfm4n\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:56.954659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.954618    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtx98\" (UniqueName: \"kubernetes.io/projected/78b1c61d-cdb6-4f3d-928d-8fc041990dad-kube-api-access-wtx98\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:56.955788 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:56.955773    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n689q\" (UniqueName: \"kubernetes.io/projected/178970c5-e1e7-4175-9613-37ef79e22aab-kube-api-access-n689q\") pod \"kube-state-metrics-54fd6bc5c4-sbx5p\" (UID: \"178970c5-e1e7-4175-9613-37ef79e22aab\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:57.004714 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.004675    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z69ss" event={"ID":"88ed25c8-f538-4c47-82a7-f41c3ec07cf4","Type":"ContainerStarted","Data":"0eb08222feaccce5780e196bbb8313514589f4452e7952d9fe9a2a92c6e419f2"}
May 29 10:13:57.005046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.004719    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z69ss" event={"ID":"88ed25c8-f538-4c47-82a7-f41c3ec07cf4","Type":"ContainerStarted","Data":"0ad1919bd6e44897ffbef5b332768d9f7247109d31363af5c048725c7ba3fa9c"}
May 29 10:13:57.120805 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.120773    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"
May 29 10:13:57.249406 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.249367    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p"]
May 29 10:13:57.252038 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:57.252012    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178970c5_e1e7_4175_9613_37ef79e22aab.slice/crio-5304680ef3cc0694afb01c578b71474f6dd3a4ead3cab3cb63beb1da7fa250ea WatchSource:0}: Error finding container 5304680ef3cc0694afb01c578b71474f6dd3a4ead3cab3cb63beb1da7fa250ea: Status 404 returned error can't find the container with id 5304680ef3cc0694afb01c578b71474f6dd3a4ead3cab3cb63beb1da7fa250ea
May 29 10:13:57.435602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.435574    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-tls\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:57.435737 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.435677    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78b1c61d-cdb6-4f3d-928d-8fc041990dad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:57.437985 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.437954    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78b1c61d-cdb6-4f3d-928d-8fc041990dad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-qs7t7\" (UID: \"78b1c61d-cdb6-4f3d-928d-8fc041990dad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:57.437985 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.437976    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e768df9-bc5a-4816-9c9e-c5df5a4d8c35-node-exporter-tls\") pod \"node-exporter-ddr92\" (UID: \"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35\") " pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:57.669510 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.669414    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5r5vj\""
May 29 10:13:57.678197 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.678168    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"
May 29 10:13:57.702903 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.702775    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jwqzg\""
May 29 10:13:57.712349 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.712322    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ddr92"
May 29 10:13:57.722967 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:57.722941    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e768df9_bc5a_4816_9c9e_c5df5a4d8c35.slice/crio-b649f51d16c036d5b252a6a9151c9bfdeef10694d98fee02ddd5c1ad0ccb75d8 WatchSource:0}: Error finding container b649f51d16c036d5b252a6a9151c9bfdeef10694d98fee02ddd5c1ad0ccb75d8: Status 404 returned error can't find the container with id b649f51d16c036d5b252a6a9151c9bfdeef10694d98fee02ddd5c1ad0ccb75d8
May 29 10:13:57.810274 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.810228    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7"]
May 29 10:13:57.812850 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:57.812823    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b1c61d_cdb6_4f3d_928d_8fc041990dad.slice/crio-8d3174b24ca283cf2725c0c25fa697fe44405ab0aaf55e05a78672cb65c13cd1 WatchSource:0}: Error finding container 8d3174b24ca283cf2725c0c25fa697fe44405ab0aaf55e05a78672cb65c13cd1: Status 404 returned error can't find the container with id 8d3174b24ca283cf2725c0c25fa697fe44405ab0aaf55e05a78672cb65c13cd1
May 29 10:13:57.826329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.826307    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 29 10:13:57.829126 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.829106    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.831005 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.830985    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 29 10:13:57.831357 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.831331    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 29 10:13:57.831530 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.831510    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gg9zm\""
May 29 10:13:57.831610 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.831510    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 29 10:13:57.832000 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.831979    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 29 10:13:57.832078 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.832019    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 29 10:13:57.832078 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.832039    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 29 10:13:57.832078 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.832065    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 29 10:13:57.832358 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.832321    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 29 10:13:57.832438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.832321    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 29 10:13:57.838370 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838453 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838513 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838556 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838594 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-web-config\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838624 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838688 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmgw\" (UniqueName: \"kubernetes.io/projected/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-kube-api-access-7kmgw\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838731 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838783 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838835 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838835 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838821    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838942 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.838998 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.838966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-config-out\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.845733 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.845709    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 29 10:13:57.940194 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.940168    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.940320 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.940211    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.940320 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.940254    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.940320 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.940286    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.940472 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.940335    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.940472 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.940374    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-config-out\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.940472 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:57.940402    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-alertmanager-trusted-ca-bundle podName:c9bec710-e704-4f60-9a51-4c0f5f42fd2d nodeName:}" failed. No retries permitted until 2026-05-29 10:13:58.440377551 +0000 UTC m=+61.336937167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c9bec710-e704-4f60-9a51-4c0f5f42fd2d") : configmap references non-existent config key: ca-bundle.crt
May 29 10:13:57.940660 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:57.940525    2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 29 10:13:57.940660 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:57.940581    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-main-tls podName:c9bec710-e704-4f60-9a51-4c0f5f42fd2d nodeName:}" failed. No retries permitted until 2026-05-29 10:13:58.440565228 +0000 UTC m=+61.337124840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c9bec710-e704-4f60-9a51-4c0f5f42fd2d") : secret "alertmanager-main-tls" not found
May 29 10:13:57.941290 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.941262    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.941450 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.941411    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.941515 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.941461    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.941574 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.941513    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.941574 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.941543    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.941699 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.941582    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-web-config\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.941699 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.941624    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.941699 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.941680    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmgw\" (UniqueName: \"kubernetes.io/projected/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-kube-api-access-7kmgw\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.942678 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.942607    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.943781 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.943737    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.943896 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.943869    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-config-out\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.943993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.943972    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.945265 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.945215    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.946227 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.946204    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.946955 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.946908    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-web-config\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.947486 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.947461    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.947793 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.947774    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:57.949941 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:57.949765    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmgw\" (UniqueName: \"kubernetes.io/projected/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-kube-api-access-7kmgw\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:58.009759 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:58.009697    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z69ss" event={"ID":"88ed25c8-f538-4c47-82a7-f41c3ec07cf4","Type":"ContainerStarted","Data":"17051166549f4b414dfad70916d163421d6a9f64024858644a913c2e1d8ecbea"}
May 29 10:13:58.011060 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:58.011014    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ddr92" event={"ID":"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35","Type":"ContainerStarted","Data":"b649f51d16c036d5b252a6a9151c9bfdeef10694d98fee02ddd5c1ad0ccb75d8"}
May 29 10:13:58.012280 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:58.012246    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p" event={"ID":"178970c5-e1e7-4175-9613-37ef79e22aab","Type":"ContainerStarted","Data":"5304680ef3cc0694afb01c578b71474f6dd3a4ead3cab3cb63beb1da7fa250ea"}
May 29 10:13:58.014985 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:58.014963    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7" event={"ID":"78b1c61d-cdb6-4f3d-928d-8fc041990dad","Type":"ContainerStarted","Data":"7de98112cc93bcd0a02eedbfdb3f61732fb3cf7cc7a3ac9019a451d53e0cbb6b"}
May 29 10:13:58.015105 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:58.014992    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7" event={"ID":"78b1c61d-cdb6-4f3d-928d-8fc041990dad","Type":"ContainerStarted","Data":"a313b16622fd7595865b4af09ea3f7224a8358167e09c88e02c29b983796c99b"}
May 29 10:13:58.015105 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:58.015005    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7" event={"ID":"78b1c61d-cdb6-4f3d-928d-8fc041990dad","Type":"ContainerStarted","Data":"8d3174b24ca283cf2725c0c25fa697fe44405ab0aaf55e05a78672cb65c13cd1"}
May 29 10:13:58.446523 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:58.446490    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:58.446772 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:58.446544    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:58.446772 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:58.446696    2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 29 10:13:58.446772 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:13:58.446770    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-main-tls podName:c9bec710-e704-4f60-9a51-4c0f5f42fd2d nodeName:}" failed. No retries permitted until 2026-05-29 10:13:59.446749689 +0000 UTC m=+62.343309291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c9bec710-e704-4f60-9a51-4c0f5f42fd2d") : secret "alertmanager-main-tls" not found
May 29 10:13:58.447479 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:58.447447    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:59.457888 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.457848    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:59.460606 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.460577    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9bec710-e704-4f60-9a51-4c0f5f42fd2d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9bec710-e704-4f60-9a51-4c0f5f42fd2d\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:59.659211 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.659179    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 29 10:13:59.858840 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.858812    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"]
May 29 10:13:59.862513 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.862493    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:13:59.864747 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.864723    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 29 10:13:59.864988 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.864904    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 29 10:13:59.865083 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.865064    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fcrsl\""
May 29 10:13:59.865330 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.865173    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 29 10:13:59.865330 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.865214    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 29 10:13:59.865330 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.865268    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 29 10:13:59.865526 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.865413    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aqart7hn9bo46\""
May 29 10:13:59.870271 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.870029    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 29 10:13:59.875336 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:13:59.875307    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bec710_e704_4f60_9a51_4c0f5f42fd2d.slice/crio-c5d8d7441023c2623e9db3b243bcc1a71b86e2ecfc3188c7ed265f253519aa33 WatchSource:0}: Error finding container c5d8d7441023c2623e9db3b243bcc1a71b86e2ecfc3188c7ed265f253519aa33: Status 404 returned error can't find the container with id c5d8d7441023c2623e9db3b243bcc1a71b86e2ecfc3188c7ed265f253519aa33
May 29 10:13:59.880881 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.880575    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"]
May 29 10:13:59.961575 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.961510    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-tls\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:13:59.961575 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.961564    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:13:59.961785 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.961606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:13:59.961785 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.961625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-metrics-client-ca\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:13:59.961785 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.961672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-grpc-tls\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:13:59.961785 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.961747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxlwf\" (UniqueName: \"kubernetes.io/projected/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-kube-api-access-fxlwf\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:13:59.961785 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.961778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:13:59.962049 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.961824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:13:59.979822 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:13:59.979790    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qxnsw"
May 29 10:14:00.024261 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.024214    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z69ss" event={"ID":"88ed25c8-f538-4c47-82a7-f41c3ec07cf4","Type":"ContainerStarted","Data":"aedee1c78391c383bff87849895ebfbbc730d5c40db9547243ef69bea16a07ed"}
May 29 10:14:00.025954 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.025920    2562 generic.go:358] "Generic (PLEG): container finished" podID="5e768df9-bc5a-4816-9c9e-c5df5a4d8c35" containerID="4c15b84a16736135ab94d07b42be0eb1c65e0ee17cdd37a20f8d609424cbb959" exitCode=0
May 29 10:14:00.026089 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.026002    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ddr92" event={"ID":"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35","Type":"ContainerDied","Data":"4c15b84a16736135ab94d07b42be0eb1c65e0ee17cdd37a20f8d609424cbb959"}
May 29 10:14:00.028955 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.028918    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p" event={"ID":"178970c5-e1e7-4175-9613-37ef79e22aab","Type":"ContainerStarted","Data":"450202c5c968d7bcef5f11dd034016be321a00dd61eaa8b2a26803e7837883e8"}
May 29 10:14:00.029119 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.029105    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p" event={"ID":"178970c5-e1e7-4175-9613-37ef79e22aab","Type":"ContainerStarted","Data":"84c5b41e306b8e1fe8ed6c9943bb90aaaaecd569797b124f58d0a1fb441bfaec"}
May 29 10:14:00.029215 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.029204    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p" event={"ID":"178970c5-e1e7-4175-9613-37ef79e22aab","Type":"ContainerStarted","Data":"a94589c80230b14aea5686391be75a2f5e8e5b02e46412434b064dccedfeaeb3"}
May 29 10:14:00.030609 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.030587    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9bec710-e704-4f60-9a51-4c0f5f42fd2d","Type":"ContainerStarted","Data":"c5d8d7441023c2623e9db3b243bcc1a71b86e2ecfc3188c7ed265f253519aa33"}
May 29 10:14:00.034471 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.034372    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7" event={"ID":"78b1c61d-cdb6-4f3d-928d-8fc041990dad","Type":"ContainerStarted","Data":"f2cb316e34bc24972bda002b3b5037c0757cfe8e32a3b01357141dc11bece517"}
May 29 10:14:00.043943 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.043886    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z69ss" podStartSLOduration=3.109429222 podStartE2EDuration="6.043868252s" podCreationTimestamp="2026-05-29 10:13:54 +0000 UTC" firstStartedPulling="2026-05-29 10:13:56.772591969 +0000 UTC m=+59.669151565" lastFinishedPulling="2026-05-29 10:13:59.707030998 +0000 UTC m=+62.603590595" observedRunningTime="2026-05-29 10:14:00.04344008 +0000 UTC m=+62.939999701" watchObservedRunningTime="2026-05-29 10:14:00.043868252 +0000 UTC m=+62.940427874"
May 29 10:14:00.063290 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.063248    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.063608 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.063585    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.063702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.063624    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-metrics-client-ca\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.063702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.063680    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-grpc-tls\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.063885 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.063864    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fxlwf\" (UniqueName: \"kubernetes.io/projected/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-kube-api-access-fxlwf\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.064273 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.064245    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.064364 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.064272    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-metrics-client-ca\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.064364 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.064348    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.064488 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.064472    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-tls\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.067191 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.067145    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-grpc-tls\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.067971 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.067948    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.068075 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.067993    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.068127 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.068081    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.068314 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.068298    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-tls\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.068645 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.068610    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.081590 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.081566    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxlwf\" (UniqueName: \"kubernetes.io/projected/ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6-kube-api-access-fxlwf\") pod \"thanos-querier-b7d97cd7c-6tffh\" (UID: \"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6\") " pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.107678 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.107043    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-qs7t7" podStartSLOduration=2.331246324 podStartE2EDuration="4.107024957s" podCreationTimestamp="2026-05-29 10:13:56 +0000 UTC" firstStartedPulling="2026-05-29 10:13:57.931181713 +0000 UTC m=+60.827741319" lastFinishedPulling="2026-05-29 10:13:59.706960337 +0000 UTC m=+62.603519952" observedRunningTime="2026-05-29 10:14:00.106489847 +0000 UTC m=+63.003049466" watchObservedRunningTime="2026-05-29 10:14:00.107024957 +0000 UTC m=+63.003584578"
May 29 10:14:00.135771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.135726    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sbx5p" podStartSLOduration=1.6828473430000002 podStartE2EDuration="4.135711233s" podCreationTimestamp="2026-05-29 10:13:56 +0000 UTC" firstStartedPulling="2026-05-29 10:13:57.254160582 +0000 UTC m=+60.150720184" lastFinishedPulling="2026-05-29 10:13:59.707024472 +0000 UTC m=+62.603584074" observedRunningTime="2026-05-29 10:14:00.134686286 +0000 UTC m=+63.031245906" watchObservedRunningTime="2026-05-29 10:14:00.135711233 +0000 UTC m=+63.032270851"
May 29 10:14:00.174834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.174798    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:00.311672 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:00.311645    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"]
May 29 10:14:00.313910 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:14:00.313882    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8f7f20_2ba8_497b_9f40_3b9c28b37ac6.slice/crio-8f4aa48db4fac52886125b9dd550ac0c31fe1dad12ab4153d4f676c7461e3a96 WatchSource:0}: Error finding container 8f4aa48db4fac52886125b9dd550ac0c31fe1dad12ab4153d4f676c7461e3a96: Status 404 returned error can't find the container with id 8f4aa48db4fac52886125b9dd550ac0c31fe1dad12ab4153d4f676c7461e3a96
May 29 10:14:01.039146 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.039111    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ddr92" event={"ID":"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35","Type":"ContainerStarted","Data":"df2c08c6d6621fac4bdeceb75582c7604f275df0f72524d7da7fbfff2883d405"}
May 29 10:14:01.039576 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.039156    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ddr92" event={"ID":"5e768df9-bc5a-4816-9c9e-c5df5a4d8c35","Type":"ContainerStarted","Data":"7eadecc536b10349ed484ad0702ff2df8bfa5e52ac2669da77b7043125a76163"}
May 29 10:14:01.040399 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.040227    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh" event={"ID":"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6","Type":"ContainerStarted","Data":"8f4aa48db4fac52886125b9dd550ac0c31fe1dad12ab4153d4f676c7461e3a96"}
May 29 10:14:01.062581 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.062533    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ddr92" podStartSLOduration=3.080152962 podStartE2EDuration="5.062492535s" podCreationTimestamp="2026-05-29 10:13:56 +0000 UTC" firstStartedPulling="2026-05-29 10:13:57.724624649 +0000 UTC m=+60.621184247" lastFinishedPulling="2026-05-29 10:13:59.706964205 +0000 UTC m=+62.603523820" observedRunningTime="2026-05-29 10:14:01.061794559 +0000 UTC m=+63.958354180" watchObservedRunningTime="2026-05-29 10:14:01.062492535 +0000 UTC m=+63.959052155"
May 29 10:14:01.164869 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.164826    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-797bb99f5d-sf4b2"]
May 29 10:14:01.167276 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.167246    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.170402 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.170359    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3vq4rrt8tmk5d\""
May 29 10:14:01.170508 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.170463    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 29 10:14:01.170793 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.170773    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vdcm9\""
May 29 10:14:01.170891 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.170823    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 29 10:14:01.170891 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.170862    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 29 10:14:01.170891 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.170877    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 29 10:14:01.179717 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.179694    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-797bb99f5d-sf4b2"]
May 29 10:14:01.279737 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.279694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db23805f-285b-4e55-8a4b-ab229fe6e126-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.279920 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.279744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/db23805f-285b-4e55-8a4b-ab229fe6e126-secret-metrics-server-client-certs\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.279920 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.279777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/db23805f-285b-4e55-8a4b-ab229fe6e126-secret-metrics-server-tls\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.279920 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.279814    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/db23805f-285b-4e55-8a4b-ab229fe6e126-metrics-server-audit-profiles\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.280033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.279919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27zfc\" (UniqueName: \"kubernetes.io/projected/db23805f-285b-4e55-8a4b-ab229fe6e126-kube-api-access-27zfc\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.280033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.279953    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/db23805f-285b-4e55-8a4b-ab229fe6e126-audit-log\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.280033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.279999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/db23805f-285b-4e55-8a4b-ab229fe6e126-client-ca-bundle\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.381364 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.381331    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db23805f-285b-4e55-8a4b-ab229fe6e126-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.381364 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.381370    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/db23805f-285b-4e55-8a4b-ab229fe6e126-secret-metrics-server-client-certs\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.381621 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.381392    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/db23805f-285b-4e55-8a4b-ab229fe6e126-secret-metrics-server-tls\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.381621 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.381435    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/db23805f-285b-4e55-8a4b-ab229fe6e126-metrics-server-audit-profiles\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.381900 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.381874    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-27zfc\" (UniqueName: \"kubernetes.io/projected/db23805f-285b-4e55-8a4b-ab229fe6e126-kube-api-access-27zfc\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.381995 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.381942    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/db23805f-285b-4e55-8a4b-ab229fe6e126-audit-log\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.382052 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.382019    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/db23805f-285b-4e55-8a4b-ab229fe6e126-client-ca-bundle\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.382271 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.382224    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db23805f-285b-4e55-8a4b-ab229fe6e126-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.382480 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.382428    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/db23805f-285b-4e55-8a4b-ab229fe6e126-audit-log\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.382480 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.382496    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/db23805f-285b-4e55-8a4b-ab229fe6e126-metrics-server-audit-profiles\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.384708 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.384685    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/db23805f-285b-4e55-8a4b-ab229fe6e126-secret-metrics-server-client-certs\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.384806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.384753    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/db23805f-285b-4e55-8a4b-ab229fe6e126-client-ca-bundle\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.384874 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.384855    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/db23805f-285b-4e55-8a4b-ab229fe6e126-secret-metrics-server-tls\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.391951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.391932    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27zfc\" (UniqueName: \"kubernetes.io/projected/db23805f-285b-4e55-8a4b-ab229fe6e126-kube-api-access-27zfc\") pod \"metrics-server-797bb99f5d-sf4b2\" (UID: \"db23805f-285b-4e55-8a4b-ab229fe6e126\") " pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.480061 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.479925    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:01.533570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.533532    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-745f644d49-wvjtt"]
May 29 10:14:01.540666 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.540547    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.554830 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.553227    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745f644d49-wvjtt"]
May 29 10:14:01.565514 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.564298    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"]
May 29 10:14:01.567136 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.566607    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"
May 29 10:14:01.571162 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.571138    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"]
May 29 10:14:01.572405 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.571433    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 29 10:14:01.572933 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.571487    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-qrqrw\""
May 29 10:14:01.588033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.588003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-config\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.588170 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.588042    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-service-ca\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.588170 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.588102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8dr\" (UniqueName: \"kubernetes.io/projected/a0b6d761-095f-456a-9c43-64ab51f22e3c-kube-api-access-hc8dr\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.588170 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.588153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-oauth-config\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.588326 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.588195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-serving-cert\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.588326 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.588213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-trusted-ca-bundle\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.588326 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.588229    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-oauth-serving-cert\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.672201 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.672129    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-797bb99f5d-sf4b2"]
May 29 10:14:01.675568 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:14:01.675535    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb23805f_285b_4e55_8a4b_ab229fe6e126.slice/crio-d82664e5fdd0227d4f4cf7c07b8285a04865aa440db3a4a759480c23ecae8e1b WatchSource:0}: Error finding container d82664e5fdd0227d4f4cf7c07b8285a04865aa440db3a4a759480c23ecae8e1b: Status 404 returned error can't find the container with id d82664e5fdd0227d4f4cf7c07b8285a04865aa440db3a4a759480c23ecae8e1b
May 29 10:14:01.689092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689070    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-trusted-ca-bundle\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689097    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-oauth-serving-cert\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689261 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689239    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-config\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689313 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689297    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-service-ca\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689427 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c45c144-f74e-441f-877d-ee7a7ebafaec-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-57r4k\" (UID: \"8c45c144-f74e-441f-877d-ee7a7ebafaec\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"
May 29 10:14:01.689542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689517    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8dr\" (UniqueName: \"kubernetes.io/projected/a0b6d761-095f-456a-9c43-64ab51f22e3c-kube-api-access-hc8dr\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689990 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689619    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-oauth-config\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689990 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689674    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-serving-cert\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689990 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689711    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-oauth-serving-cert\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689990 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689946    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-trusted-ca-bundle\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689990 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689950    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-config\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.689990 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.689975    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-service-ca\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.692134 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.692112    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-serving-cert\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.692200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.692146    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-oauth-config\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.697592 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.697503    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8dr\" (UniqueName: \"kubernetes.io/projected/a0b6d761-095f-456a-9c43-64ab51f22e3c-kube-api-access-hc8dr\") pod \"console-745f644d49-wvjtt\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") " pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.790505 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.790461    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c45c144-f74e-441f-877d-ee7a7ebafaec-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-57r4k\" (UID: \"8c45c144-f74e-441f-877d-ee7a7ebafaec\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"
May 29 10:14:01.793091 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.793060    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c45c144-f74e-441f-877d-ee7a7ebafaec-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-57r4k\" (UID: \"8c45c144-f74e-441f-877d-ee7a7ebafaec\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"
May 29 10:14:01.887250 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.887183    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:01.896111 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:01.896081    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"
May 29 10:14:02.043963 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.043887    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2" event={"ID":"db23805f-285b-4e55-8a4b-ab229fe6e126","Type":"ContainerStarted","Data":"d82664e5fdd0227d4f4cf7c07b8285a04865aa440db3a4a759480c23ecae8e1b"}
May 29 10:14:02.045503 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.045475    2562 generic.go:358] "Generic (PLEG): container finished" podID="c9bec710-e704-4f60-9a51-4c0f5f42fd2d" containerID="a777e3a966430871f1235fb7628efed1d202b16c559fb06063442abcad25da6b" exitCode=0
May 29 10:14:02.045596 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.045561    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9bec710-e704-4f60-9a51-4c0f5f42fd2d","Type":"ContainerDied","Data":"a777e3a966430871f1235fb7628efed1d202b16c559fb06063442abcad25da6b"}
May 29 10:14:02.506256 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.498275    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:14:02.507527 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.507292    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 29 10:14:02.512233 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.512179    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad608802-fd8d-49b6-945f-b56237be7996-metrics-certs\") pod \"network-metrics-daemon-wk82g\" (UID: \"ad608802-fd8d-49b6-945f-b56237be7996\") " pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:14:02.512748 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.512715    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745f644d49-wvjtt"]
May 29 10:14:02.515943 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:14:02.515918    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b6d761_095f_456a_9c43_64ab51f22e3c.slice/crio-e134d61d340573bf1659a4093e09df522e4f6cb9abba30ac45bf13fc77ae9f9e WatchSource:0}: Error finding container e134d61d340573bf1659a4093e09df522e4f6cb9abba30ac45bf13fc77ae9f9e: Status 404 returned error can't find the container with id e134d61d340573bf1659a4093e09df522e4f6cb9abba30ac45bf13fc77ae9f9e
May 29 10:14:02.540947 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.540892    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"]
May 29 10:14:02.544085 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:14:02.544056    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c45c144_f74e_441f_877d_ee7a7ebafaec.slice/crio-83b6705b9e4d7c9fd5b13bb673502d19839b713e8d2ea10e3cf7e8db2807c54e WatchSource:0}: Error finding container 83b6705b9e4d7c9fd5b13bb673502d19839b713e8d2ea10e3cf7e8db2807c54e: Status 404 returned error can't find the container with id 83b6705b9e4d7c9fd5b13bb673502d19839b713e8d2ea10e3cf7e8db2807c54e
May 29 10:14:02.599742 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.599707    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:14:02.602241 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.602215    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 29 10:14:02.612139 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.612123    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 29 10:14:02.623095 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.623075    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjkg\" (UniqueName: \"kubernetes.io/projected/eb12e862-d565-46d8-b7da-df8d75835853-kube-api-access-9gjkg\") pod \"network-check-target-2j2z5\" (UID: \"eb12e862-d565-46d8-b7da-df8d75835853\") " pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:14:02.813994 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.813910    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hmwm9\""
May 29 10:14:02.819868 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.819838    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cf54m\""
May 29 10:14:02.822345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.822322    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wk82g"
May 29 10:14:02.828588 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:02.828372    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:14:03.035235 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.035039    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wk82g"]
May 29 10:14:03.038743 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:14:03.038711    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad608802_fd8d_49b6_945f_b56237be7996.slice/crio-84feade5dc28c436157ca2380489a45eac916dd993bcca1d98ff5bf027d2cff3 WatchSource:0}: Error finding container 84feade5dc28c436157ca2380489a45eac916dd993bcca1d98ff5bf027d2cff3: Status 404 returned error can't find the container with id 84feade5dc28c436157ca2380489a45eac916dd993bcca1d98ff5bf027d2cff3
May 29 10:14:03.050952 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.050757    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k" event={"ID":"8c45c144-f74e-441f-877d-ee7a7ebafaec","Type":"ContainerStarted","Data":"83b6705b9e4d7c9fd5b13bb673502d19839b713e8d2ea10e3cf7e8db2807c54e"}
May 29 10:14:03.054610 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.054576    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh" event={"ID":"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6","Type":"ContainerStarted","Data":"3bc4d1120bbfabf75de3958430ec50c671221a112f05a6a2043bd8cf637269de"}
May 29 10:14:03.054735 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.054622    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh" event={"ID":"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6","Type":"ContainerStarted","Data":"2e793cf5e739b37c39676eb9497e1ad191dae199e7e656f9c9fd6cc97f8e87b2"}
May 29 10:14:03.054735 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.054660    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh" event={"ID":"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6","Type":"ContainerStarted","Data":"e5c5ccdb3d1432eb5ddf233d285a2056b9542b731d5111deb571ffcc2e85ed8e"}
May 29 10:14:03.056772 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.056727    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wk82g" event={"ID":"ad608802-fd8d-49b6-945f-b56237be7996","Type":"ContainerStarted","Data":"84feade5dc28c436157ca2380489a45eac916dd993bcca1d98ff5bf027d2cff3"}
May 29 10:14:03.058614 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.058589    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745f644d49-wvjtt" event={"ID":"a0b6d761-095f-456a-9c43-64ab51f22e3c","Type":"ContainerStarted","Data":"b7a529800b29c278db71fe79c014f2f1524bd0c8bca2eefa6c7c18df93f9d8df"}
May 29 10:14:03.058733 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.058619    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745f644d49-wvjtt" event={"ID":"a0b6d761-095f-456a-9c43-64ab51f22e3c","Type":"ContainerStarted","Data":"e134d61d340573bf1659a4093e09df522e4f6cb9abba30ac45bf13fc77ae9f9e"}
May 29 10:14:03.063328 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.063304    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2j2z5"]
May 29 10:14:03.067359 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:14:03.067331    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb12e862_d565_46d8_b7da_df8d75835853.slice/crio-d21c11d9b6905094d6225cfcde12857443383f8219a4eae719cf9a4aed78784a WatchSource:0}: Error finding container d21c11d9b6905094d6225cfcde12857443383f8219a4eae719cf9a4aed78784a: Status 404 returned error can't find the container with id d21c11d9b6905094d6225cfcde12857443383f8219a4eae719cf9a4aed78784a
May 29 10:14:03.093733 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.093168    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-745f644d49-wvjtt" podStartSLOduration=2.093150177 podStartE2EDuration="2.093150177s" podCreationTimestamp="2026-05-29 10:14:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:14:03.091273937 +0000 UTC m=+65.987833579" watchObservedRunningTime="2026-05-29 10:14:03.093150177 +0000 UTC m=+65.989709797"
May 29 10:14:03.108331 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.108301    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 29 10:14:03.112181 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.112156    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.116288 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116259    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 29 10:14:03.117423 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116314    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 29 10:14:03.117564 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116362    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 29 10:14:03.117729 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116381    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 29 10:14:03.117910 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116508    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 29 10:14:03.118015 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116523    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 29 10:14:03.118088 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.118070    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-64mqq\""
May 29 10:14:03.118153 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116713    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cue1ekek9dv2\""
May 29 10:14:03.118208 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116765    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 29 10:14:03.118208 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116857    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 29 10:14:03.118309 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116977    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 29 10:14:03.118390 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.116897    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 29 10:14:03.120367 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.120349    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 29 10:14:03.132044 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.128300    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 29 10:14:03.132044 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.128892    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 29 10:14:03.208570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f59f6b1-1862-42c8-b20c-aecb86a65037-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-config\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208692    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f59f6b1-1862-42c8-b20c-aecb86a65037-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.208834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.209329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.209329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208888    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f59f6b1-1862-42c8-b20c-aecb86a65037-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.209329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zktpb\" (UniqueName: \"kubernetes.io/projected/7f59f6b1-1862-42c8-b20c-aecb86a65037-kube-api-access-zktpb\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.209329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.209329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.208967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.209329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.209006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.209329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.209038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.309850 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.309809    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.309860    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f59f6b1-1862-42c8-b20c-aecb86a65037-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.309892    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zktpb\" (UniqueName: \"kubernetes.io/projected/7f59f6b1-1862-42c8-b20c-aecb86a65037-kube-api-access-zktpb\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.309921    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.309959    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310037    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310069    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310137    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310175    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f59f6b1-1862-42c8-b20c-aecb86a65037-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310200    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310226    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310524 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310253    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-config\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310524 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310325    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310524 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310382    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f59f6b1-1862-42c8-b20c-aecb86a65037-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310524 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310439    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310524 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310477    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310524 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310512    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.310839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.310544    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.314352 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.312584    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.314352 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.313944    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.314352 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.314127    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.319031 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.318922    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f59f6b1-1862-42c8-b20c-aecb86a65037-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.320239 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.319545    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-config\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.320239 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.319584    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.320239 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.319784    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.320239 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.320120    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.320587 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.320545    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f59f6b1-1862-42c8-b20c-aecb86a65037-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.322212 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.322166    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.322768 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.322357    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f59f6b1-1862-42c8-b20c-aecb86a65037-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.322768 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.322436    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.322768 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.322722    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f59f6b1-1862-42c8-b20c-aecb86a65037-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.322961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.322857    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.323750 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.323712    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.325588 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.325309    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.325588 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.325544    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f59f6b1-1862-42c8-b20c-aecb86a65037-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.328117 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.328075    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zktpb\" (UniqueName: \"kubernetes.io/projected/7f59f6b1-1862-42c8-b20c-aecb86a65037-kube-api-access-zktpb\") pod \"prometheus-k8s-0\" (UID: \"7f59f6b1-1862-42c8-b20c-aecb86a65037\") " pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:03.440719 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:03.440525    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:04.062970 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:04.062931    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2j2z5" event={"ID":"eb12e862-d565-46d8-b7da-df8d75835853","Type":"ContainerStarted","Data":"d21c11d9b6905094d6225cfcde12857443383f8219a4eae719cf9a4aed78784a"}
May 29 10:14:05.342364 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:05.342332    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 29 10:14:05.351686 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:14:05.350576    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f59f6b1_1862_42c8_b20c_aecb86a65037.slice/crio-9d5f72bc70248673df346e8788fd0fac1b9b588f9b6483d313fea42db2c4f318 WatchSource:0}: Error finding container 9d5f72bc70248673df346e8788fd0fac1b9b588f9b6483d313fea42db2c4f318: Status 404 returned error can't find the container with id 9d5f72bc70248673df346e8788fd0fac1b9b588f9b6483d313fea42db2c4f318
May 29 10:14:06.074282 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.074195    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9bec710-e704-4f60-9a51-4c0f5f42fd2d","Type":"ContainerStarted","Data":"5f55df1845c9ad5c306e34b830694121c3ba7bb1a5abac1400db43e519ff7e3b"}
May 29 10:14:06.074282 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.074243    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9bec710-e704-4f60-9a51-4c0f5f42fd2d","Type":"ContainerStarted","Data":"399e384759d1a241c5c983bd331a00cc300976af43489e3ba25c903d4391cd8c"}
May 29 10:14:06.074479 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.074258    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9bec710-e704-4f60-9a51-4c0f5f42fd2d","Type":"ContainerStarted","Data":"a6f9950daf38182686dbaaa885e2967fcef36e5b36d79b05124f07125e8061d7"}
May 29 10:14:06.074479 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.074301    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9bec710-e704-4f60-9a51-4c0f5f42fd2d","Type":"ContainerStarted","Data":"2ca37f1bf868e9cc143fba4abc61374a341b620ddece381e79bac9647ec0dc6d"}
May 29 10:14:06.074479 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.074314    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9bec710-e704-4f60-9a51-4c0f5f42fd2d","Type":"ContainerStarted","Data":"43f511f43a6c9426c8864d9131f1071dcf525c8ab766041574c7722efb1da29c"}
May 29 10:14:06.074479 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.074325    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9bec710-e704-4f60-9a51-4c0f5f42fd2d","Type":"ContainerStarted","Data":"21e43bb15e9ac62925a7fbd653d7b261833fe7409e311462929bcf1261861b46"}
May 29 10:14:06.077542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.077504    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh" event={"ID":"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6","Type":"ContainerStarted","Data":"3e9793461decd70066db6279b35492c637c8a7576432c65cffadeb24cace67ca"}
May 29 10:14:06.077542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.077542    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh" event={"ID":"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6","Type":"ContainerStarted","Data":"041102efc50d9906cc2287603beed7e1b4cdc60ebee3107bde5aa22bc5f8058f"}
May 29 10:14:06.077870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.077557    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh" event={"ID":"ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6","Type":"ContainerStarted","Data":"9e6578d576137276cce65a9b50aaf7858614cf25f7d491aeed367156631e0144"}
May 29 10:14:06.078358 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.078336    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:06.080433 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.080383    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wk82g" event={"ID":"ad608802-fd8d-49b6-945f-b56237be7996","Type":"ContainerStarted","Data":"50b23d55856b829308ffc2b294ca91eff1e6dcd07134d8740c13b4d8a989d9a0"}
May 29 10:14:06.080433 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.080414    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wk82g" event={"ID":"ad608802-fd8d-49b6-945f-b56237be7996","Type":"ContainerStarted","Data":"b0dce9042bd793e48fdef9cc0b13ba7f12f12a573b136d86583ba3dd5f237e4e"}
May 29 10:14:06.082237 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.082214    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2" event={"ID":"db23805f-285b-4e55-8a4b-ab229fe6e126","Type":"ContainerStarted","Data":"874abf032e058c6b5eacc494e26053e47bfc6ad77f7ad4ab41bf5718f3f5dd64"}
May 29 10:14:06.083849 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.083755    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f59f6b1-1862-42c8-b20c-aecb86a65037" containerID="13536f64dce9854c97bba3a6dec67b2836777d195d3cdd22abcd9d84487b0483" exitCode=0
May 29 10:14:06.083849 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.083838    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f59f6b1-1862-42c8-b20c-aecb86a65037","Type":"ContainerDied","Data":"13536f64dce9854c97bba3a6dec67b2836777d195d3cdd22abcd9d84487b0483"}
May 29 10:14:06.083963 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.083869    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f59f6b1-1862-42c8-b20c-aecb86a65037","Type":"ContainerStarted","Data":"9d5f72bc70248673df346e8788fd0fac1b9b588f9b6483d313fea42db2c4f318"}
May 29 10:14:06.085936 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.085914    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k" event={"ID":"8c45c144-f74e-441f-877d-ee7a7ebafaec","Type":"ContainerStarted","Data":"50481b2bf5d6bc7a801aa01ed794c95d69f3934e02937ae9038e09c24b8b12d3"}
May 29 10:14:06.086310 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.086290    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"
May 29 10:14:06.092530 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.092510    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k"
May 29 10:14:06.102823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.101915    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.813467224 podStartE2EDuration="9.101899309s" podCreationTimestamp="2026-05-29 10:13:57 +0000 UTC" firstStartedPulling="2026-05-29 10:13:59.879224458 +0000 UTC m=+62.775784055" lastFinishedPulling="2026-05-29 10:14:05.167656538 +0000 UTC m=+68.064216140" observedRunningTime="2026-05-29 10:14:06.100361082 +0000 UTC m=+68.996920718" watchObservedRunningTime="2026-05-29 10:14:06.101899309 +0000 UTC m=+68.998458928"
May 29 10:14:06.122480 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.122424    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wk82g" podStartSLOduration=66.968497897 podStartE2EDuration="1m9.122410522s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:14:03.043220433 +0000 UTC m=+65.939780034" lastFinishedPulling="2026-05-29 10:14:05.197133062 +0000 UTC m=+68.093692659" observedRunningTime="2026-05-29 10:14:06.120754559 +0000 UTC m=+69.017314179" watchObservedRunningTime="2026-05-29 10:14:06.122410522 +0000 UTC m=+69.018970140"
May 29 10:14:06.138870 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.138816    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-57r4k" podStartSLOduration=2.495968903 podStartE2EDuration="5.138797823s" podCreationTimestamp="2026-05-29 10:14:01 +0000 UTC" firstStartedPulling="2026-05-29 10:14:02.546575959 +0000 UTC m=+65.443135556" lastFinishedPulling="2026-05-29 10:14:05.189404864 +0000 UTC m=+68.085964476" observedRunningTime="2026-05-29 10:14:06.134525348 +0000 UTC m=+69.031084984" watchObservedRunningTime="2026-05-29 10:14:06.138797823 +0000 UTC m=+69.035357443"
May 29 10:14:06.161138 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.161085    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh" podStartSLOduration=2.310495093 podStartE2EDuration="7.161070415s" podCreationTimestamp="2026-05-29 10:13:59 +0000 UTC" firstStartedPulling="2026-05-29 10:14:00.315909994 +0000 UTC m=+63.212469590" lastFinishedPulling="2026-05-29 10:14:05.166485311 +0000 UTC m=+68.063044912" observedRunningTime="2026-05-29 10:14:06.159675902 +0000 UTC m=+69.056235518" watchObservedRunningTime="2026-05-29 10:14:06.161070415 +0000 UTC m=+69.057630034"
May 29 10:14:06.215687 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:06.215609    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2" podStartSLOduration=1.7261550300000001 podStartE2EDuration="5.215587791s" podCreationTimestamp="2026-05-29 10:14:01 +0000 UTC" firstStartedPulling="2026-05-29 10:14:01.678187291 +0000 UTC m=+64.574747091" lastFinishedPulling="2026-05-29 10:14:05.167620254 +0000 UTC m=+68.064179852" observedRunningTime="2026-05-29 10:14:06.214508628 +0000 UTC m=+69.111068248" watchObservedRunningTime="2026-05-29 10:14:06.215587791 +0000 UTC m=+69.112147414"
May 29 10:14:07.091059 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:07.091012    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2j2z5" event={"ID":"eb12e862-d565-46d8-b7da-df8d75835853","Type":"ContainerStarted","Data":"d491690155adfbeef21f6be545166830a18c46f6288247211d44ce12bc54b4c3"}
May 29 10:14:07.091722 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:07.091663    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:14:07.099274 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:07.099250    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b7d97cd7c-6tffh"
May 29 10:14:07.109953 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:07.109913    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2j2z5" podStartSLOduration=66.339066758 podStartE2EDuration="1m10.109898753s" podCreationTimestamp="2026-05-29 10:12:57 +0000 UTC" firstStartedPulling="2026-05-29 10:14:03.072266029 +0000 UTC m=+65.968825633" lastFinishedPulling="2026-05-29 10:14:06.843098028 +0000 UTC m=+69.739657628" observedRunningTime="2026-05-29 10:14:07.107409539 +0000 UTC m=+70.003969160" watchObservedRunningTime="2026-05-29 10:14:07.109898753 +0000 UTC m=+70.006458374"
May 29 10:14:10.104517 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:10.104439    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f59f6b1-1862-42c8-b20c-aecb86a65037","Type":"ContainerStarted","Data":"e13efe4c90be862dacbe4e9bc4efbd34612fcbd3a916971b08fafbbf29b4e352"}
May 29 10:14:10.104517 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:10.104480    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f59f6b1-1862-42c8-b20c-aecb86a65037","Type":"ContainerStarted","Data":"43259362fbc2f8639b90d7094da0d93b100a0d82953e670f3a74f8e8a44c7770"}
May 29 10:14:10.104517 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:10.104493    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f59f6b1-1862-42c8-b20c-aecb86a65037","Type":"ContainerStarted","Data":"68a6e9ff08559b2c3cf23e405e4154c0672c7379644ab3cf338012a15ffb884e"}
May 29 10:14:10.104517 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:10.104502    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f59f6b1-1862-42c8-b20c-aecb86a65037","Type":"ContainerStarted","Data":"858b2eac55360c4395ce46b0c0523a76e2616757fd74ac9e88a0e965d754ecf2"}
May 29 10:14:10.104517 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:10.104510    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f59f6b1-1862-42c8-b20c-aecb86a65037","Type":"ContainerStarted","Data":"354325236c2f03e63dc9d9919f6fb8e34c6ff4651a32137af9588a63d3b16766"}
May 29 10:14:10.104517 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:10.104518    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f59f6b1-1862-42c8-b20c-aecb86a65037","Type":"ContainerStarted","Data":"b5a662e9ff50f22c06c2f4de4b364bd92ff5736d67a174a648e7e70cc979afb6"}
May 29 10:14:10.136733 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:10.136670    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.625533725 podStartE2EDuration="7.136626484s" podCreationTimestamp="2026-05-29 10:14:03 +0000 UTC" firstStartedPulling="2026-05-29 10:14:06.08505471 +0000 UTC m=+68.981614307" lastFinishedPulling="2026-05-29 10:14:09.596147466 +0000 UTC m=+72.492707066" observedRunningTime="2026-05-29 10:14:10.131866607 +0000 UTC m=+73.028426228" watchObservedRunningTime="2026-05-29 10:14:10.136626484 +0000 UTC m=+73.033186104"
May 29 10:14:11.887575 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:11.887419    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:11.887575 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:11.887468    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:11.892312 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:11.892288    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:12.115684 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:12.115656    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:14:12.161690 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:12.161594    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ffd4fc95-jtqxf"]
May 29 10:14:13.441820 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:13.441789    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:14:21.480839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:21.480796    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:21.480839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:21.480840    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:37.181461 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.181404    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7ffd4fc95-jtqxf" podUID="8d23793b-1b40-4b96-bcdd-3a224b3bfefc" containerName="console" containerID="cri-o://404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e" gracePeriod=15
May 29 10:14:37.497706 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.497679    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7ffd4fc95-jtqxf_8d23793b-1b40-4b96-bcdd-3a224b3bfefc/console/0.log"
May 29 10:14:37.497829 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.497765    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:14:37.519565 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.519537    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-config\") pod \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") "
May 29 10:14:37.519746 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.519625    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-trusted-ca-bundle\") pod \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") "
May 29 10:14:37.519746 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.519680    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-oauth-serving-cert\") pod \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") "
May 29 10:14:37.519746 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.519726    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-oauth-config\") pod \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") "
May 29 10:14:37.519746 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.519746    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-service-ca\") pod \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") "
May 29 10:14:37.519931 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.519772    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvm2z\" (UniqueName: \"kubernetes.io/projected/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-kube-api-access-fvm2z\") pod \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") "
May 29 10:14:37.519931 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.519797    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-serving-cert\") pod \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\" (UID: \"8d23793b-1b40-4b96-bcdd-3a224b3bfefc\") "
May 29 10:14:37.520213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.520041    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8d23793b-1b40-4b96-bcdd-3a224b3bfefc" (UID: "8d23793b-1b40-4b96-bcdd-3a224b3bfefc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:14:37.520213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.520057    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-config" (OuterVolumeSpecName: "console-config") pod "8d23793b-1b40-4b96-bcdd-3a224b3bfefc" (UID: "8d23793b-1b40-4b96-bcdd-3a224b3bfefc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:14:37.520213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.520162    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8d23793b-1b40-4b96-bcdd-3a224b3bfefc" (UID: "8d23793b-1b40-4b96-bcdd-3a224b3bfefc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:14:37.520213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.520175    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-service-ca" (OuterVolumeSpecName: "service-ca") pod "8d23793b-1b40-4b96-bcdd-3a224b3bfefc" (UID: "8d23793b-1b40-4b96-bcdd-3a224b3bfefc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:14:37.522138 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.522098    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8d23793b-1b40-4b96-bcdd-3a224b3bfefc" (UID: "8d23793b-1b40-4b96-bcdd-3a224b3bfefc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:14:37.522242 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.522221    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8d23793b-1b40-4b96-bcdd-3a224b3bfefc" (UID: "8d23793b-1b40-4b96-bcdd-3a224b3bfefc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:14:37.522343 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.522322    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-kube-api-access-fvm2z" (OuterVolumeSpecName: "kube-api-access-fvm2z") pod "8d23793b-1b40-4b96-bcdd-3a224b3bfefc" (UID: "8d23793b-1b40-4b96-bcdd-3a224b3bfefc"). InnerVolumeSpecName "kube-api-access-fvm2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:14:37.621425 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.621381    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-oauth-config\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:14:37.621425 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.621417    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-service-ca\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:14:37.621425 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.621431    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvm2z\" (UniqueName: \"kubernetes.io/projected/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-kube-api-access-fvm2z\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:14:37.621703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.621444    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-serving-cert\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:14:37.621703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.621457    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-console-config\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:14:37.621703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.621468    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-trusted-ca-bundle\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:14:37.621703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:37.621478    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d23793b-1b40-4b96-bcdd-3a224b3bfefc-oauth-serving-cert\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:14:38.192871 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.192845    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7ffd4fc95-jtqxf_8d23793b-1b40-4b96-bcdd-3a224b3bfefc/console/0.log"
May 29 10:14:38.193253 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.192883    2562 generic.go:358] "Generic (PLEG): container finished" podID="8d23793b-1b40-4b96-bcdd-3a224b3bfefc" containerID="404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e" exitCode=2
May 29 10:14:38.193253 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.192946    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffd4fc95-jtqxf" event={"ID":"8d23793b-1b40-4b96-bcdd-3a224b3bfefc","Type":"ContainerDied","Data":"404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e"}
May 29 10:14:38.193253 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.192975    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffd4fc95-jtqxf" event={"ID":"8d23793b-1b40-4b96-bcdd-3a224b3bfefc","Type":"ContainerDied","Data":"9fb9dd7a0599429b7ff7d082ec0baf9b7fcd8c17c0d9987eee2d632d55e2f932"}
May 29 10:14:38.193253 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.192972    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffd4fc95-jtqxf"
May 29 10:14:38.193253 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.192986    2562 scope.go:117] "RemoveContainer" containerID="404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e"
May 29 10:14:38.201163 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.201142    2562 scope.go:117] "RemoveContainer" containerID="404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e"
May 29 10:14:38.201462 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:14:38.201441    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e\": container with ID starting with 404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e not found: ID does not exist" containerID="404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e"
May 29 10:14:38.201572 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.201473    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e"} err="failed to get container status \"404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e\": rpc error: code = NotFound desc = could not find container \"404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e\": container with ID starting with 404f6786cf0808bb1bc8bc11c1ad62890bbb84fb5a986711e6784fdc54f7931e not found: ID does not exist"
May 29 10:14:38.212068 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.212043    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ffd4fc95-jtqxf"]
May 29 10:14:38.215150 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:38.215119    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ffd4fc95-jtqxf"]
May 29 10:14:39.101698 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:39.101670    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2j2z5"
May 29 10:14:39.703267 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:39.703239    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d23793b-1b40-4b96-bcdd-3a224b3bfefc" path="/var/lib/kubelet/pods/8d23793b-1b40-4b96-bcdd-3a224b3bfefc/volumes"
May 29 10:14:41.485983 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:41.485949    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:14:41.489738 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:14:41.489719    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-797bb99f5d-sf4b2"
May 29 10:15:03.441517 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:03.441472    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:15:03.462649 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:03.462611    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:15:04.287317 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:04.287286    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 29 10:15:21.103597 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.103561    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"]
May 29 10:15:21.104095 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.104074    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d23793b-1b40-4b96-bcdd-3a224b3bfefc" containerName="console"
May 29 10:15:21.104164 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.104098    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d23793b-1b40-4b96-bcdd-3a224b3bfefc" containerName="console"
May 29 10:15:21.104216 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.104190    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8d23793b-1b40-4b96-bcdd-3a224b3bfefc" containerName="console"
May 29 10:15:21.107089 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.107063    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.107530 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.107499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-secret-telemeter-client\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.107668 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.107582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-telemeter-client-tls\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.107668 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.107613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a64ba713-1a70-4565-bd5c-eec6ce2774f9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.107777 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.107703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.107777 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.107761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a64ba713-1a70-4565-bd5c-eec6ce2774f9-serving-certs-ca-bundle\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.107850 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.107800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-federate-client-tls\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.107850 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.107834    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mrdd\" (UniqueName: \"kubernetes.io/projected/a64ba713-1a70-4565-bd5c-eec6ce2774f9-kube-api-access-8mrdd\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.107924 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.107901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a64ba713-1a70-4565-bd5c-eec6ce2774f9-metrics-client-ca\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.109446 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.109425    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 29 10:15:21.109446 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.109439    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 29 10:15:21.109604 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.109438    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 29 10:15:21.109604 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.109587    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7tsf2\""
May 29 10:15:21.109742 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.109725    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 29 10:15:21.109855 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.109747    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 29 10:15:21.115533 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.115512    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 29 10:15:21.118884 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.118853    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"]
May 29 10:15:21.208585 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.208549    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a64ba713-1a70-4565-bd5c-eec6ce2774f9-serving-certs-ca-bundle\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.208771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.208592    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-federate-client-tls\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.208771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.208618    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8mrdd\" (UniqueName: \"kubernetes.io/projected/a64ba713-1a70-4565-bd5c-eec6ce2774f9-kube-api-access-8mrdd\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.208771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.208663    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a64ba713-1a70-4565-bd5c-eec6ce2774f9-metrics-client-ca\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.208771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.208703    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-secret-telemeter-client\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.208771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.208722    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-telemeter-client-tls\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.208771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.208739    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a64ba713-1a70-4565-bd5c-eec6ce2774f9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.208771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.208768    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.209394 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.209365    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a64ba713-1a70-4565-bd5c-eec6ce2774f9-serving-certs-ca-bundle\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.209568 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.209383    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a64ba713-1a70-4565-bd5c-eec6ce2774f9-metrics-client-ca\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.209748 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.209727    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a64ba713-1a70-4565-bd5c-eec6ce2774f9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.211293 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.211265    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.211419 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.211398    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-federate-client-tls\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.211770 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.211750    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-secret-telemeter-client\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.211841 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.211786    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a64ba713-1a70-4565-bd5c-eec6ce2774f9-telemeter-client-tls\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.216048 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.216027    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mrdd\" (UniqueName: \"kubernetes.io/projected/a64ba713-1a70-4565-bd5c-eec6ce2774f9-kube-api-access-8mrdd\") pod \"telemeter-client-7dcb655b7b-jpcww\" (UID: \"a64ba713-1a70-4565-bd5c-eec6ce2774f9\") " pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.419237 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.419129    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"
May 29 10:15:21.551223 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:21.551195    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dcb655b7b-jpcww"]
May 29 10:15:21.552973 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:15:21.552945    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64ba713_1a70_4565_bd5c_eec6ce2774f9.slice/crio-29146e1c3f914bcafb4601342f71e04d86e548632fd4b64df59ac53ececfe5d8 WatchSource:0}: Error finding container 29146e1c3f914bcafb4601342f71e04d86e548632fd4b64df59ac53ececfe5d8: Status 404 returned error can't find the container with id 29146e1c3f914bcafb4601342f71e04d86e548632fd4b64df59ac53ececfe5d8
May 29 10:15:22.326861 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:22.326822    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww" event={"ID":"a64ba713-1a70-4565-bd5c-eec6ce2774f9","Type":"ContainerStarted","Data":"29146e1c3f914bcafb4601342f71e04d86e548632fd4b64df59ac53ececfe5d8"}
May 29 10:15:23.333146 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:23.333116    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww" event={"ID":"a64ba713-1a70-4565-bd5c-eec6ce2774f9","Type":"ContainerStarted","Data":"fd45ae3e9047f1c6675c0939c11c85f21674f8fda3d1200973d5dfba830666f4"}
May 29 10:15:23.333463 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:23.333155    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww" event={"ID":"a64ba713-1a70-4565-bd5c-eec6ce2774f9","Type":"ContainerStarted","Data":"d09df6ed5fbe77c11d5522bc98edaa6f26f502beffb7fbab60c4f878b2178ae4"}
May 29 10:15:24.338844 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:24.338800    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww" event={"ID":"a64ba713-1a70-4565-bd5c-eec6ce2774f9","Type":"ContainerStarted","Data":"b82ab6af8f22fd05d0d8aabe6ab2ef629f4dc990f98aded4fd4c8f940f18cd7a"}
May 29 10:15:24.361153 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:24.361099    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7dcb655b7b-jpcww" podStartSLOduration=1.7242919140000001 podStartE2EDuration="3.361072529s" podCreationTimestamp="2026-05-29 10:15:21 +0000 UTC" firstStartedPulling="2026-05-29 10:15:21.554743311 +0000 UTC m=+144.451302907" lastFinishedPulling="2026-05-29 10:15:23.191523923 +0000 UTC m=+146.088083522" observedRunningTime="2026-05-29 10:15:24.360457176 +0000 UTC m=+147.257016795" watchObservedRunningTime="2026-05-29 10:15:24.361072529 +0000 UTC m=+147.257632151"
May 29 10:15:42.329421 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:15:42.329344    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-745f644d49-wvjtt"]
May 29 10:16:07.088828 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.088792    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-x52b4"]
May 29 10:16:07.094183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.094162    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.099186 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.099160    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x52b4"]
May 29 10:16:07.099787 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.099768    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 29 10:16:07.125601 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.125569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7e789c0-dfb3-48d7-bd89-cbc8e59242c0-dbus\") pod \"global-pull-secret-syncer-x52b4\" (UID: \"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0\") " pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.125803 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.125620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7e789c0-dfb3-48d7-bd89-cbc8e59242c0-kubelet-config\") pod \"global-pull-secret-syncer-x52b4\" (UID: \"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0\") " pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.125803 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.125662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7e789c0-dfb3-48d7-bd89-cbc8e59242c0-original-pull-secret\") pod \"global-pull-secret-syncer-x52b4\" (UID: \"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0\") " pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.227120 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.227080    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7e789c0-dfb3-48d7-bd89-cbc8e59242c0-dbus\") pod \"global-pull-secret-syncer-x52b4\" (UID: \"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0\") " pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.227310 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.227141    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7e789c0-dfb3-48d7-bd89-cbc8e59242c0-kubelet-config\") pod \"global-pull-secret-syncer-x52b4\" (UID: \"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0\") " pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.227310 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.227167    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7e789c0-dfb3-48d7-bd89-cbc8e59242c0-original-pull-secret\") pod \"global-pull-secret-syncer-x52b4\" (UID: \"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0\") " pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.227310 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.227286    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7e789c0-dfb3-48d7-bd89-cbc8e59242c0-kubelet-config\") pod \"global-pull-secret-syncer-x52b4\" (UID: \"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0\") " pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.227433 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.227306    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7e789c0-dfb3-48d7-bd89-cbc8e59242c0-dbus\") pod \"global-pull-secret-syncer-x52b4\" (UID: \"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0\") " pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.229602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.229581    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7e789c0-dfb3-48d7-bd89-cbc8e59242c0-original-pull-secret\") pod \"global-pull-secret-syncer-x52b4\" (UID: \"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0\") " pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.354398 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.354365    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-745f644d49-wvjtt" podUID="a0b6d761-095f-456a-9c43-64ab51f22e3c" containerName="console" containerID="cri-o://b7a529800b29c278db71fe79c014f2f1524bd0c8bca2eefa6c7c18df93f9d8df" gracePeriod=15
May 29 10:16:07.404542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.404508    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x52b4"
May 29 10:16:07.473198 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.473174    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-745f644d49-wvjtt_a0b6d761-095f-456a-9c43-64ab51f22e3c/console/0.log"
May 29 10:16:07.473327 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.473216    2562 generic.go:358] "Generic (PLEG): container finished" podID="a0b6d761-095f-456a-9c43-64ab51f22e3c" containerID="b7a529800b29c278db71fe79c014f2f1524bd0c8bca2eefa6c7c18df93f9d8df" exitCode=2
May 29 10:16:07.473327 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.473288    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745f644d49-wvjtt" event={"ID":"a0b6d761-095f-456a-9c43-64ab51f22e3c","Type":"ContainerDied","Data":"b7a529800b29c278db71fe79c014f2f1524bd0c8bca2eefa6c7c18df93f9d8df"}
May 29 10:16:07.536582 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.536540    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x52b4"]
May 29 10:16:07.539134 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:16:07.539103    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e789c0_dfb3_48d7_bd89_cbc8e59242c0.slice/crio-6d3bcecec5611c2fc502e9a06ce54c59ae13268c2e4f41a306126889e6f2761d WatchSource:0}: Error finding container 6d3bcecec5611c2fc502e9a06ce54c59ae13268c2e4f41a306126889e6f2761d: Status 404 returned error can't find the container with id 6d3bcecec5611c2fc502e9a06ce54c59ae13268c2e4f41a306126889e6f2761d
May 29 10:16:07.594951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.594929    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-745f644d49-wvjtt_a0b6d761-095f-456a-9c43-64ab51f22e3c/console/0.log"
May 29 10:16:07.595071 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.594990    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:16:07.630874 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.630785    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-serving-cert\") pod \"a0b6d761-095f-456a-9c43-64ab51f22e3c\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") "
May 29 10:16:07.630874 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.630831    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-oauth-config\") pod \"a0b6d761-095f-456a-9c43-64ab51f22e3c\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") "
May 29 10:16:07.630874 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.630850    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-service-ca\") pod \"a0b6d761-095f-456a-9c43-64ab51f22e3c\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") "
May 29 10:16:07.631147 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.630900    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-config\") pod \"a0b6d761-095f-456a-9c43-64ab51f22e3c\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") "
May 29 10:16:07.631147 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.630935    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc8dr\" (UniqueName: \"kubernetes.io/projected/a0b6d761-095f-456a-9c43-64ab51f22e3c-kube-api-access-hc8dr\") pod \"a0b6d761-095f-456a-9c43-64ab51f22e3c\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") "
May 29 10:16:07.631147 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.630976    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-oauth-serving-cert\") pod \"a0b6d761-095f-456a-9c43-64ab51f22e3c\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") "
May 29 10:16:07.631147 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.631000    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-trusted-ca-bundle\") pod \"a0b6d761-095f-456a-9c43-64ab51f22e3c\" (UID: \"a0b6d761-095f-456a-9c43-64ab51f22e3c\") "
May 29 10:16:07.631347 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.631305    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-service-ca" (OuterVolumeSpecName: "service-ca") pod "a0b6d761-095f-456a-9c43-64ab51f22e3c" (UID: "a0b6d761-095f-456a-9c43-64ab51f22e3c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:16:07.631347 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.631328    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-config" (OuterVolumeSpecName: "console-config") pod "a0b6d761-095f-456a-9c43-64ab51f22e3c" (UID: "a0b6d761-095f-456a-9c43-64ab51f22e3c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:16:07.631563 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.631542    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a0b6d761-095f-456a-9c43-64ab51f22e3c" (UID: "a0b6d761-095f-456a-9c43-64ab51f22e3c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:16:07.631563 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.631539    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a0b6d761-095f-456a-9c43-64ab51f22e3c" (UID: "a0b6d761-095f-456a-9c43-64ab51f22e3c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:16:07.633358 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.633330    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a0b6d761-095f-456a-9c43-64ab51f22e3c" (UID: "a0b6d761-095f-456a-9c43-64ab51f22e3c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:16:07.633358 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.633348    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a0b6d761-095f-456a-9c43-64ab51f22e3c" (UID: "a0b6d761-095f-456a-9c43-64ab51f22e3c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:16:07.633358 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.633331    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b6d761-095f-456a-9c43-64ab51f22e3c-kube-api-access-hc8dr" (OuterVolumeSpecName: "kube-api-access-hc8dr") pod "a0b6d761-095f-456a-9c43-64ab51f22e3c" (UID: "a0b6d761-095f-456a-9c43-64ab51f22e3c"). InnerVolumeSpecName "kube-api-access-hc8dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:16:07.731912 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.731886    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8dr\" (UniqueName: \"kubernetes.io/projected/a0b6d761-095f-456a-9c43-64ab51f22e3c-kube-api-access-hc8dr\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:16:07.731912 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.731911    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-oauth-serving-cert\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:16:07.731912 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.731921    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-trusted-ca-bundle\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:16:07.732145 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.731931    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-serving-cert\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:16:07.732145 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.731947    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-oauth-config\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:16:07.732145 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.731962    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-service-ca\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:16:07.732145 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:07.731975    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0b6d761-095f-456a-9c43-64ab51f22e3c-console-config\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:16:08.478179 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:08.478150    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-745f644d49-wvjtt_a0b6d761-095f-456a-9c43-64ab51f22e3c/console/0.log"
May 29 10:16:08.478615 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:08.478271    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745f644d49-wvjtt" event={"ID":"a0b6d761-095f-456a-9c43-64ab51f22e3c","Type":"ContainerDied","Data":"e134d61d340573bf1659a4093e09df522e4f6cb9abba30ac45bf13fc77ae9f9e"}
May 29 10:16:08.478615 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:08.478312    2562 scope.go:117] "RemoveContainer" containerID="b7a529800b29c278db71fe79c014f2f1524bd0c8bca2eefa6c7c18df93f9d8df"
May 29 10:16:08.478615 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:08.478346    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745f644d49-wvjtt"
May 29 10:16:08.479999 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:08.479857    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x52b4" event={"ID":"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0","Type":"ContainerStarted","Data":"6d3bcecec5611c2fc502e9a06ce54c59ae13268c2e4f41a306126889e6f2761d"}
May 29 10:16:08.517410 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:08.517373    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-745f644d49-wvjtt"]
May 29 10:16:08.538043 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:08.538004    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-745f644d49-wvjtt"]
May 29 10:16:09.704568 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:09.704537    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0b6d761-095f-456a-9c43-64ab51f22e3c" path="/var/lib/kubelet/pods/a0b6d761-095f-456a-9c43-64ab51f22e3c/volumes"
May 29 10:16:11.491178 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:11.491091    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x52b4" event={"ID":"d7e789c0-dfb3-48d7-bd89-cbc8e59242c0","Type":"ContainerStarted","Data":"a0ec80ba813cfb611c0d9d7916ad7c1e9ac40ae48183e0381da4ee3b10940672"}
May 29 10:16:11.507310 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:16:11.507260    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x52b4" podStartSLOduration=0.812859885 podStartE2EDuration="4.507245933s" podCreationTimestamp="2026-05-29 10:16:07 +0000 UTC" firstStartedPulling="2026-05-29 10:16:07.540777296 +0000 UTC m=+190.437336893" lastFinishedPulling="2026-05-29 10:16:11.235163342 +0000 UTC m=+194.131722941" observedRunningTime="2026-05-29 10:16:11.505923363 +0000 UTC m=+194.402482984" watchObservedRunningTime="2026-05-29 10:16:11.507245933 +0000 UTC m=+194.403805552"
May 29 10:17:57.592482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:17:57.592437    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:17:57.593810 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:17:57.593773    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:18:32.612105 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.612067    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"]
May 29 10:18:32.612717 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.612398    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b6d761-095f-456a-9c43-64ab51f22e3c" containerName="console"
May 29 10:18:32.612717 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.612442    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b6d761-095f-456a-9c43-64ab51f22e3c" containerName="console"
May 29 10:18:32.612717 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.612541    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0b6d761-095f-456a-9c43-64ab51f22e3c" containerName="console"
May 29 10:18:32.616287 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.616261    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.618529 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.618503    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 29 10:18:32.618687 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.618502    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 29 10:18:32.618687 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.618562    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 29 10:18:32.618687 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.618509    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-fnplq\""
May 29 10:18:32.618864 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.618692    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 29 10:18:32.618864 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.618711    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 29 10:18:32.618976 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.618868    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 10:18:32.618976 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.618937    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 10:18:32.627618 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.627593    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"]
May 29 10:18:32.720880 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.720847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-registry-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.720880 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.720886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-offline-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.721107 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.720909    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-ui-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.721107 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.720995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-feast-data\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.721107 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.721059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-online-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.721107 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.721074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bksd\" (UniqueName: \"kubernetes.io/projected/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-kube-api-access-9bksd\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.822525 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.822489    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-online-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.822753 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.822533    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9bksd\" (UniqueName: \"kubernetes.io/projected/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-kube-api-access-9bksd\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.822753 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.822575    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-registry-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.822753 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.822604    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-offline-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.822753 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.822676    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-ui-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.823000 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.822795    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-feast-data\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.823193 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.823156    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-feast-data\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.825240 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.825218    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-online-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.825346 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.825248    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-registry-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.825346 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.825302    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-ui-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.825425 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.825409    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-offline-tls\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.830900 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.830879    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bksd\" (UniqueName: \"kubernetes.io/projected/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-kube-api-access-9bksd\") pod \"feast-simple-feast-setup-754cfff778-crcn8\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:32.934794 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:32.934700    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:18:33.073420 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:33.069817    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"]
May 29 10:18:33.075302 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:18:33.075272    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d6ef90a_e2ee_48a2_8f01_a4aee88f4f72.slice/crio-294e7cc3704c38576f096df1cb52386c282872124f5ea1326f7cdf151ec851d8 WatchSource:0}: Error finding container 294e7cc3704c38576f096df1cb52386c282872124f5ea1326f7cdf151ec851d8: Status 404 returned error can't find the container with id 294e7cc3704c38576f096df1cb52386c282872124f5ea1326f7cdf151ec851d8
May 29 10:18:33.076942 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:33.076924    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 10:18:33.919822 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:18:33.919743    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerStarted","Data":"294e7cc3704c38576f096df1cb52386c282872124f5ea1326f7cdf151ec851d8"}
May 29 10:19:07.024525 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:07.024487    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerStarted","Data":"c2bbf9880d6d0e33c402002e6a13cde7952d5399a1dbe353aea155d2bd09a39c"}
May 29 10:19:11.037519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:11.037482    2562 generic.go:358] "Generic (PLEG): container finished" podID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerID="c2bbf9880d6d0e33c402002e6a13cde7952d5399a1dbe353aea155d2bd09a39c" exitCode=0
May 29 10:19:11.037907 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:11.037549    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerDied","Data":"c2bbf9880d6d0e33c402002e6a13cde7952d5399a1dbe353aea155d2bd09a39c"}
May 29 10:19:12.042658 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:12.042600    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerStarted","Data":"77110969233ab4f9605ed4be736b4082592f9eb69f2f76e67d80d8950409ce7f"}
May 29 10:19:17.061063 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:17.061016    2562 generic.go:358] "Generic (PLEG): container finished" podID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerID="77110969233ab4f9605ed4be736b4082592f9eb69f2f76e67d80d8950409ce7f" exitCode=0
May 29 10:19:17.061463 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:17.061084    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerDied","Data":"77110969233ab4f9605ed4be736b4082592f9eb69f2f76e67d80d8950409ce7f"}
May 29 10:19:17.070914 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:17.070895    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 29 10:19:18.069046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:18.068996    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerStarted","Data":"5ed0bb05b701b419506da3f464712eca7d9ebc0537fa0369ffe56a6621b554ed"}
May 29 10:19:18.069046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:18.069043    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerStarted","Data":"6a3500985018f9194878bdd56216568f2d32cb93d97bab25d7030d05a51df5d1"}
May 29 10:19:18.069787 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:18.069058    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerStarted","Data":"372e1503ee357799307a4f8dbefa35000c7e893837556af1f40291a5aeae114b"}
May 29 10:19:18.069787 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:18.069072    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerStarted","Data":"2a61c96b0cd5b78561d26fe821bfcbbf10d6b8f483808c12a46fc14fff888d9c"}
May 29 10:19:18.095868 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:18.095805    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podStartSLOduration=12.860824917 podStartE2EDuration="46.095783783s" podCreationTimestamp="2026-05-29 10:18:32 +0000 UTC" firstStartedPulling="2026-05-29 10:18:33.077074447 +0000 UTC m=+335.973634044" lastFinishedPulling="2026-05-29 10:19:06.31203331 +0000 UTC m=+369.208592910" observedRunningTime="2026-05-29 10:19:18.093938574 +0000 UTC m=+380.990498194" watchObservedRunningTime="2026-05-29 10:19:18.095783783 +0000 UTC m=+380.992343404"
May 29 10:19:20.935486 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:20.935442    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:20.935987 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:20.935498    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:20.935987 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:20.935511    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:20.935987 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:20.935521    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:20.937415 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:20.937367    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused"
May 29 10:19:20.937415 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:20.937402    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused"
May 29 10:19:20.937597 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:20.937442    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.21:8816: connect: connection refused"
May 29 10:19:20.937597 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:20.937393    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused"
May 29 10:19:22.935301 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:22.935255    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:22.935720 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:22.935314    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:22.935720 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:22.935326    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:22.935720 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:22.935333    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:23.935658 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:23.935605    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:23.936119 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:23.936032    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:23.936217 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:23.936204    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:23.936834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:23.936780    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:23.936957 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:23.936851    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:23.937165 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:23.937144    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:23.940926 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:23.940905    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:19:24.098509 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:19:24.098485    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:20:25.979012 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:25.978922    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"]
May 29 10:20:25.979538 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:25.979375    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="registry" containerID="cri-o://2a61c96b0cd5b78561d26fe821bfcbbf10d6b8f483808c12a46fc14fff888d9c" gracePeriod=30
May 29 10:20:25.979538 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:25.979484    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="offline" containerID="cri-o://6a3500985018f9194878bdd56216568f2d32cb93d97bab25d7030d05a51df5d1" gracePeriod=30
May 29 10:20:25.979538 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:25.979531    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="online" containerID="cri-o://372e1503ee357799307a4f8dbefa35000c7e893837556af1f40291a5aeae114b" gracePeriod=30
May 29 10:20:25.979732 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:25.979516    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="ui" containerID="cri-o://5ed0bb05b701b419506da3f464712eca7d9ebc0537fa0369ffe56a6621b554ed" gracePeriod=30
May 29 10:20:26.283112 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.283019    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"]
May 29 10:20:26.286554 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.286532    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.288848 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.288825    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-54lmm\""
May 29 10:20:26.298507 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.298476    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"]
May 29 10:20:26.356705 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.356667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-registry-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.356705 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.356709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-offline-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.356977 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.356736    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zwg\" (UniqueName: \"kubernetes.io/projected/d27cc8ca-b289-4599-b149-538ad7b12e85-kube-api-access-v9zwg\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.356977 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.356903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d27cc8ca-b289-4599-b149-538ad7b12e85-feast-data\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.356977 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.356950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-ui-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.357179 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.356980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-online-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.457904 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.457833    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d27cc8ca-b289-4599-b149-538ad7b12e85-feast-data\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.457904 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.457869    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-ui-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.458033 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:20:26.457969    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 29 10:20:26.458033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.458007    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-online-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.458130 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:20:26.458059    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-ui-tls podName:d27cc8ca-b289-4599-b149-538ad7b12e85 nodeName:}" failed. No retries permitted until 2026-05-29 10:20:26.958037668 +0000 UTC m=+449.854597282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-ui-tls") pod "feast-simple-feast-setup-754cfff778-wxng8" (UID: "d27cc8ca-b289-4599-b149-538ad7b12e85") : secret "feast-simple-feast-setup-ui-tls" not found
May 29 10:20:26.458130 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:20:26.458111    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 29 10:20:26.458130 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.458122    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-registry-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.458293 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.458158    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-offline-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.458293 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:20:26.458180    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-online-tls podName:d27cc8ca-b289-4599-b149-538ad7b12e85 nodeName:}" failed. No retries permitted until 2026-05-29 10:20:26.958162619 +0000 UTC m=+449.854722225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-online-tls") pod "feast-simple-feast-setup-754cfff778-wxng8" (UID: "d27cc8ca-b289-4599-b149-538ad7b12e85") : secret "feast-simple-feast-setup-online-tls" not found
May 29 10:20:26.458293 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.458232    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zwg\" (UniqueName: \"kubernetes.io/projected/d27cc8ca-b289-4599-b149-538ad7b12e85-kube-api-access-v9zwg\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.458442 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.458322    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d27cc8ca-b289-4599-b149-538ad7b12e85-feast-data\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.460898 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.460870    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-registry-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.460898 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.460883    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-offline-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.467179 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.467158    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zwg\" (UniqueName: \"kubernetes.io/projected/d27cc8ca-b289-4599-b149-538ad7b12e85-kube-api-access-v9zwg\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.962626 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.962543    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-ui-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.962626 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.962581    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-online-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.964973 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.964947    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-online-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:26.965079 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:26.965003    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-ui-tls\") pod \"feast-simple-feast-setup-754cfff778-wxng8\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") " pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:27.196950 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:27.196909    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:27.299125 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:27.299093    2562 generic.go:358] "Generic (PLEG): container finished" podID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerID="5ed0bb05b701b419506da3f464712eca7d9ebc0537fa0369ffe56a6621b554ed" exitCode=0
May 29 10:20:27.299125 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:27.299122    2562 generic.go:358] "Generic (PLEG): container finished" podID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerID="372e1503ee357799307a4f8dbefa35000c7e893837556af1f40291a5aeae114b" exitCode=0
May 29 10:20:27.299297 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:27.299154    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerDied","Data":"5ed0bb05b701b419506da3f464712eca7d9ebc0537fa0369ffe56a6621b554ed"}
May 29 10:20:27.299297 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:27.299178    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerDied","Data":"372e1503ee357799307a4f8dbefa35000c7e893837556af1f40291a5aeae114b"}
May 29 10:20:27.332908 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:27.332872    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"]
May 29 10:20:27.335109 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:20:27.335081    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd27cc8ca_b289_4599_b149_538ad7b12e85.slice/crio-655a7554f63bdc450dd21c786936651ec27d79677a28c9a5aaf302f31e60a745 WatchSource:0}: Error finding container 655a7554f63bdc450dd21c786936651ec27d79677a28c9a5aaf302f31e60a745: Status 404 returned error can't find the container with id 655a7554f63bdc450dd21c786936651ec27d79677a28c9a5aaf302f31e60a745
May 29 10:20:28.303033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:28.302990    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerStarted","Data":"a9d5a283c81296feb8e4c021de66aff728c9d1c0ee26a825ad0810adfd00a0c1"}
May 29 10:20:28.303410 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:28.303041    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerStarted","Data":"655a7554f63bdc450dd21c786936651ec27d79677a28c9a5aaf302f31e60a745"}
May 29 10:20:32.316437 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:32.316350    2562 generic.go:358] "Generic (PLEG): container finished" podID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerID="a9d5a283c81296feb8e4c021de66aff728c9d1c0ee26a825ad0810adfd00a0c1" exitCode=0
May 29 10:20:32.316437 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:32.316410    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerDied","Data":"a9d5a283c81296feb8e4c021de66aff728c9d1c0ee26a825ad0810adfd00a0c1"}
May 29 10:20:33.321422 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:33.321382    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerStarted","Data":"17bf4396668b1fcc1f86792bb4ea1623d084433b49c04c26bbcba3589a161008"}
May 29 10:20:33.937607 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:33.937553    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused"
May 29 10:20:34.096594 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:34.096542    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused"
May 29 10:20:38.338588 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:38.338510    2562 generic.go:358] "Generic (PLEG): container finished" podID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerID="17bf4396668b1fcc1f86792bb4ea1623d084433b49c04c26bbcba3589a161008" exitCode=0
May 29 10:20:38.338948 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:38.338582    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerDied","Data":"17bf4396668b1fcc1f86792bb4ea1623d084433b49c04c26bbcba3589a161008"}
May 29 10:20:39.346708 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:39.346662    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerStarted","Data":"c438738e7fdd69cf88f1b26ac94308c461bcf1bd8761f8ba455bbbc9b0eccb4b"}
May 29 10:20:39.346708 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:39.346714    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerStarted","Data":"d0a85e27518b1467b4ea5c826184db3bd3e5603e7639ad78d668aa9aa675fed2"}
May 29 10:20:39.347256 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:39.346731    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerStarted","Data":"12d20bedb94a2c9c85d70f4682bb460a218c36c00654c6fc1997847f3f55eb1a"}
May 29 10:20:39.347256 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:39.346744    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerStarted","Data":"f8f67f2cd40cf391a080db92592f71b07af9cca7f52c2759eba02fddcc0924e7"}
May 29 10:20:39.372552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:39.372490    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podStartSLOduration=13.372471201 podStartE2EDuration="13.372471201s" podCreationTimestamp="2026-05-29 10:20:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:20:39.369775695 +0000 UTC m=+462.266335315" watchObservedRunningTime="2026-05-29 10:20:39.372471201 +0000 UTC m=+462.269030822"
May 29 10:20:42.197949 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:42.197907    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:42.197949 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:42.197962    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:42.198577 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:42.197977    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:42.198577 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:42.198432    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:42.200244 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:42.200212    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused"
May 29 10:20:42.200452 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:42.200239    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 29 10:20:42.200573 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:42.200262    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused"
May 29 10:20:42.200573 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:42.200301    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 29 10:20:43.937183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:43.937135    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused"
May 29 10:20:44.096060 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:44.096017    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused"
May 29 10:20:45.198395 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.198365    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.198822 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.198492    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.198822 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.198748    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.199126 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.199102    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.199227 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.199134    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.199227 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.199147    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.199227 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.199205    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.199362 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.199342    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.199614 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.199597    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.202767 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.202749    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.368378 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.368351    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:45.371349 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:45.371315    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:20:53.937470 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:53.937428    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused"
May 29 10:20:53.937977 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:53.937592    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:20:54.096699 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:54.096627    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused"
May 29 10:20:54.096883 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:54.096810    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:20:56.404414 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.404375    2562 generic.go:358] "Generic (PLEG): container finished" podID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerID="6a3500985018f9194878bdd56216568f2d32cb93d97bab25d7030d05a51df5d1" exitCode=137
May 29 10:20:56.404414 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.404398    2562 generic.go:358] "Generic (PLEG): container finished" podID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerID="2a61c96b0cd5b78561d26fe821bfcbbf10d6b8f483808c12a46fc14fff888d9c" exitCode=137
May 29 10:20:56.404816 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.404445    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerDied","Data":"6a3500985018f9194878bdd56216568f2d32cb93d97bab25d7030d05a51df5d1"}
May 29 10:20:56.404816 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.404478    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerDied","Data":"2a61c96b0cd5b78561d26fe821bfcbbf10d6b8f483808c12a46fc14fff888d9c"}
May 29 10:20:56.635008 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.634987    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:20:56.728583 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.728503    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bksd\" (UniqueName: \"kubernetes.io/projected/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-kube-api-access-9bksd\") pod \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") "
May 29 10:20:56.728583 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.728569    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-ui-tls\") pod \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") "
May 29 10:20:56.728809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.728657    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-feast-data\") pod \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") "
May 29 10:20:56.728809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.728680    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-registry-tls\") pod \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") "
May 29 10:20:56.728809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.728701    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-offline-tls\") pod \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") "
May 29 10:20:56.728809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.728715    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-online-tls\") pod \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\" (UID: \"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72\") "
May 29 10:20:56.729486 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.729410    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-feast-data" (OuterVolumeSpecName: "feast-data") pod "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" (UID: "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:20:56.731140 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.731107    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" (UID: "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:20:56.731249 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.731162    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" (UID: "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:20:56.731249 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.731166    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-kube-api-access-9bksd" (OuterVolumeSpecName: "kube-api-access-9bksd") pod "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" (UID: "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72"). InnerVolumeSpecName "kube-api-access-9bksd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:20:56.731364 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.731341    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-online-tls" (OuterVolumeSpecName: "online-tls") pod "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" (UID: "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:20:56.731364 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.731354    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" (UID: "1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:20:56.830148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.830111    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:20:56.830148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.830142    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:20:56.830148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.830150    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:20:56.830148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.830158    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:20:56.830471 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.830167    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bksd\" (UniqueName: \"kubernetes.io/projected/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-kube-api-access-9bksd\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:20:56.830471 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:56.830183    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72-ui-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:20:57.409894 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.409862    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8" event={"ID":"1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72","Type":"ContainerDied","Data":"294e7cc3704c38576f096df1cb52386c282872124f5ea1326f7cdf151ec851d8"}
May 29 10:20:57.409894 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.409886    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"
May 29 10:20:57.410358 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.409907    2562 scope.go:117] "RemoveContainer" containerID="5ed0bb05b701b419506da3f464712eca7d9ebc0537fa0369ffe56a6621b554ed"
May 29 10:20:57.419217 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.419200    2562 scope.go:117] "RemoveContainer" containerID="6a3500985018f9194878bdd56216568f2d32cb93d97bab25d7030d05a51df5d1"
May 29 10:20:57.427199 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.427179    2562 scope.go:117] "RemoveContainer" containerID="372e1503ee357799307a4f8dbefa35000c7e893837556af1f40291a5aeae114b"
May 29 10:20:57.438188 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.438154    2562 scope.go:117] "RemoveContainer" containerID="2a61c96b0cd5b78561d26fe821bfcbbf10d6b8f483808c12a46fc14fff888d9c"
May 29 10:20:57.438315 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.438293    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"]
May 29 10:20:57.439991 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.439967    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-crcn8"]
May 29 10:20:57.446248 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.446219    2562 scope.go:117] "RemoveContainer" containerID="77110969233ab4f9605ed4be736b4082592f9eb69f2f76e67d80d8950409ce7f"
May 29 10:20:57.453320 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.453302    2562 scope.go:117] "RemoveContainer" containerID="c2bbf9880d6d0e33c402002e6a13cde7952d5399a1dbe353aea155d2bd09a39c"
May 29 10:20:57.704185 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:20:57.704104    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" path="/var/lib/kubelet/pods/1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72/volumes"
May 29 10:21:48.749661 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749561    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"]
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749916    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="feast-init"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749927    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="feast-init"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749942    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="online"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749947    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="online"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749954    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="ui"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749959    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="ui"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749977    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="feast-apply"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749982    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="feast-apply"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749991    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="offline"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.749996    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="offline"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.750003    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="registry"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.750008    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="registry"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.750060    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="registry"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.750067    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="offline"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.750075    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="ui"
May 29 10:21:48.750213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.750083    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="1d6ef90a-e2ee-48a2-8f01-a4aee88f4f72" containerName="online"
May 29 10:21:48.753167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.753151    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.757413 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.757391    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 29 10:21:48.757557 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.757482    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 29 10:21:48.757557 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.757499    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 29 10:21:48.757682 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.757623    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 29 10:21:48.761193 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.761173    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-l6vn5\""
May 29 10:21:48.761589 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.761571    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 29 10:21:48.761746 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.761731    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 29 10:21:48.761849 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.761832    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 29 10:21:48.785382 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.785353    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"]
May 29 10:21:48.797945 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.797919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/70a4f017-1b14-4308-a9cc-f2967b41b607-feast-data\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.798054 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.797956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdql\" (UniqueName: \"kubernetes.io/projected/70a4f017-1b14-4308-a9cc-f2967b41b607-kube-api-access-dxdql\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.798054 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.798030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-offline-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.798282 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.798253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-ui-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.798413 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.798316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/70a4f017-1b14-4308-a9cc-f2967b41b607-registry-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.798413 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.798349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-online-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.899732 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.899693    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-offline-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.899924 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.899769    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-ui-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.899924 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.899808    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/70a4f017-1b14-4308-a9cc-f2967b41b607-registry-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.899924 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:21:48.899830    2562 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found
May 29 10:21:48.899924 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:21:48.899923    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-offline-tls podName:70a4f017-1b14-4308-a9cc-f2967b41b607 nodeName:}" failed. No retries permitted until 2026-05-29 10:21:49.399900227 +0000 UTC m=+532.296459824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-offline-tls") pod "feast-simple-feast-remote-setup-6c9df695d5-r24k9" (UID: "70a4f017-1b14-4308-a9cc-f2967b41b607") : secret "feast-simple-feast-remote-setup-offline-tls" not found
May 29 10:21:48.900202 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.899839    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-online-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.900202 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.899991    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/70a4f017-1b14-4308-a9cc-f2967b41b607-feast-data\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.900202 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.900027    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdql\" (UniqueName: \"kubernetes.io/projected/70a4f017-1b14-4308-a9cc-f2967b41b607-kube-api-access-dxdql\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.900334 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.900319    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/70a4f017-1b14-4308-a9cc-f2967b41b607-feast-data\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.900777 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.900758    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/70a4f017-1b14-4308-a9cc-f2967b41b607-registry-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.902360 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.902339    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-online-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.902471 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.902456    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-ui-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:48.914925 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:48.914903    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdql\" (UniqueName: \"kubernetes.io/projected/70a4f017-1b14-4308-a9cc-f2967b41b607-kube-api-access-dxdql\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:49.407351 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:49.407313    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-offline-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:49.409879 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:49.409847    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-offline-tls\") pod \"feast-simple-feast-remote-setup-6c9df695d5-r24k9\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:49.662352 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:49.662264    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:21:49.813654 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:49.813608    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"]
May 29 10:21:49.818308 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:21:49.818280    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70a4f017_1b14_4308_a9cc_f2967b41b607.slice/crio-76c6fbe82fa940ac567c0eb5753222fefce772fe88ee48d04bd8cc71bcfc04d3 WatchSource:0}: Error finding container 76c6fbe82fa940ac567c0eb5753222fefce772fe88ee48d04bd8cc71bcfc04d3: Status 404 returned error can't find the container with id 76c6fbe82fa940ac567c0eb5753222fefce772fe88ee48d04bd8cc71bcfc04d3
May 29 10:21:50.574266 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:50.574232    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerStarted","Data":"42a9b01a4397020b700be782f67952f9b073e54536989caf16a678cc62956844"}
May 29 10:21:50.574266 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:50.574272    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerStarted","Data":"76c6fbe82fa940ac567c0eb5753222fefce772fe88ee48d04bd8cc71bcfc04d3"}
May 29 10:21:54.586835 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:54.586797    2562 generic.go:358] "Generic (PLEG): container finished" podID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerID="42a9b01a4397020b700be782f67952f9b073e54536989caf16a678cc62956844" exitCode=0
May 29 10:21:54.587248 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:54.586875    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerDied","Data":"42a9b01a4397020b700be782f67952f9b073e54536989caf16a678cc62956844"}
May 29 10:21:55.591451 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:21:55.591415    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerStarted","Data":"e1a8e60c5ff3c91d3eea3470fb39b2585bc6b5461f74c61f0e4aaac7c0f718e0"}
May 29 10:22:00.607740 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:00.607707    2562 generic.go:358] "Generic (PLEG): container finished" podID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerID="e1a8e60c5ff3c91d3eea3470fb39b2585bc6b5461f74c61f0e4aaac7c0f718e0" exitCode=0
May 29 10:22:00.608109 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:00.607780    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerDied","Data":"e1a8e60c5ff3c91d3eea3470fb39b2585bc6b5461f74c61f0e4aaac7c0f718e0"}
May 29 10:22:00.612839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:00.612818    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 29 10:22:01.614656 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.614593    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerStarted","Data":"83aad4186d86107984548ba316145eedd6200d5087fde3815b3635824d7f70bb"}
May 29 10:22:01.615022 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.614668    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerStarted","Data":"944a13800627a89c8ba2449c85625782a5ce46643d9e9674ef2b4a7668e44bd9"}
May 29 10:22:01.615022 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.614705    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerStarted","Data":"4dc4f90b04561b5d2128f1ab5a975330fe3c91be0f69656ffe09d03caf2c9f09"}
May 29 10:22:01.663443 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.663394    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:01.663443 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.663443    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:01.663689 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.663458    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:01.665465 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.665419    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused"
May 29 10:22:01.665604 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.665469    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 29 10:22:01.665604 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.665483    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 29 10:22:01.701704 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:01.701650    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podStartSLOduration=13.701621704 podStartE2EDuration="13.701621704s" podCreationTimestamp="2026-05-29 10:21:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:22:01.690043688 +0000 UTC m=+544.586603308" watchObservedRunningTime="2026-05-29 10:22:01.701621704 +0000 UTC m=+544.598181323"
May 29 10:22:04.663006 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:04.662948    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 29 10:22:04.663423 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:04.662948    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused"
May 29 10:22:04.663423 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:04.662952    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 29 10:22:07.664048 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:07.664009    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:07.664500 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:07.664149    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:07.664662 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:07.664621    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:07.664733 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:07.664669    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:07.665064 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:07.665047    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:07.665131 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:07.665119    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:07.668373 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:07.668357    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:08.638564 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:08.638529    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:08.641579 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:08.641560    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:22:57.617532 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:57.617507    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:22:57.619263 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:22:57.619235    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:23:11.397193 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:11.397110    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"]
May 29 10:23:11.397882 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:11.397547    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="online" containerID="cri-o://4dc4f90b04561b5d2128f1ab5a975330fe3c91be0f69656ffe09d03caf2c9f09" gracePeriod=30
May 29 10:23:11.397882 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:11.397626    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="offline" containerID="cri-o://944a13800627a89c8ba2449c85625782a5ce46643d9e9674ef2b4a7668e44bd9" gracePeriod=30
May 29 10:23:11.398056 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:11.397923    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="ui" containerID="cri-o://83aad4186d86107984548ba316145eedd6200d5087fde3815b3635824d7f70bb" gracePeriod=30
May 29 10:23:11.514263 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:11.514210    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"]
May 29 10:23:11.514772 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:11.514557    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="registry" containerID="cri-o://f8f67f2cd40cf391a080db92592f71b07af9cca7f52c2759eba02fddcc0924e7" gracePeriod=30
May 29 10:23:11.514772 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:11.514624    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="offline" containerID="cri-o://d0a85e27518b1467b4ea5c826184db3bd3e5603e7639ad78d668aa9aa675fed2" gracePeriod=30
May 29 10:23:11.514772 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:11.514659    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="online" containerID="cri-o://12d20bedb94a2c9c85d70f4682bb460a218c36c00654c6fc1997847f3f55eb1a" gracePeriod=30
May 29 10:23:11.514772 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:11.514619    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="ui" containerID="cri-o://c438738e7fdd69cf88f1b26ac94308c461bcf1bd8761f8ba455bbbc9b0eccb4b" gracePeriod=30
May 29 10:23:12.839258 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:12.839226    2562 generic.go:358] "Generic (PLEG): container finished" podID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerID="83aad4186d86107984548ba316145eedd6200d5087fde3815b3635824d7f70bb" exitCode=0
May 29 10:23:12.839258 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:12.839250    2562 generic.go:358] "Generic (PLEG): container finished" podID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerID="4dc4f90b04561b5d2128f1ab5a975330fe3c91be0f69656ffe09d03caf2c9f09" exitCode=0
May 29 10:23:12.839739 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:12.839297    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerDied","Data":"83aad4186d86107984548ba316145eedd6200d5087fde3815b3635824d7f70bb"}
May 29 10:23:12.839739 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:12.839330    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerDied","Data":"4dc4f90b04561b5d2128f1ab5a975330fe3c91be0f69656ffe09d03caf2c9f09"}
May 29 10:23:12.841695 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:12.841675    2562 generic.go:358] "Generic (PLEG): container finished" podID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerID="c438738e7fdd69cf88f1b26ac94308c461bcf1bd8761f8ba455bbbc9b0eccb4b" exitCode=0
May 29 10:23:12.841695 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:12.841693    2562 generic.go:358] "Generic (PLEG): container finished" podID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerID="12d20bedb94a2c9c85d70f4682bb460a218c36c00654c6fc1997847f3f55eb1a" exitCode=0
May 29 10:23:12.841839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:12.841741    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerDied","Data":"c438738e7fdd69cf88f1b26ac94308c461bcf1bd8761f8ba455bbbc9b0eccb4b"}
May 29 10:23:12.841839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:12.841767    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerDied","Data":"12d20bedb94a2c9c85d70f4682bb460a218c36c00654c6fc1997847f3f55eb1a"}
May 29 10:23:15.199227 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:15.199180    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 29 10:23:15.368555 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:15.368507    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 29 10:23:17.665523 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:17.665470    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 29 10:23:18.639494 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:18.639450    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 29 10:23:25.199407 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:25.199348    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 29 10:23:25.368929 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:25.368886    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 29 10:23:27.665342 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:27.665291    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 29 10:23:28.638765 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:28.638717    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 29 10:23:35.199624 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:35.199571    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 29 10:23:35.201091 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:35.199761    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:23:35.369527 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:35.369479    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 29 10:23:35.369735 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:35.369676    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:23:37.665480 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:37.665427    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 29 10:23:37.665910 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:37.665563    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:23:38.639791 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:38.639750    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 29 10:23:38.639980 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:38.639884    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:23:41.931707 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:41.931670    2562 generic.go:358] "Generic (PLEG): container finished" podID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerID="944a13800627a89c8ba2449c85625782a5ce46643d9e9674ef2b4a7668e44bd9" exitCode=137
May 29 10:23:41.932118 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:41.931741    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerDied","Data":"944a13800627a89c8ba2449c85625782a5ce46643d9e9674ef2b4a7668e44bd9"}
May 29 10:23:41.934693 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:41.934667    2562 generic.go:358] "Generic (PLEG): container finished" podID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerID="d0a85e27518b1467b4ea5c826184db3bd3e5603e7639ad78d668aa9aa675fed2" exitCode=137
May 29 10:23:41.934693 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:41.934688    2562 generic.go:358] "Generic (PLEG): container finished" podID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerID="f8f67f2cd40cf391a080db92592f71b07af9cca7f52c2759eba02fddcc0924e7" exitCode=137
May 29 10:23:41.934852 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:41.934736    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerDied","Data":"d0a85e27518b1467b4ea5c826184db3bd3e5603e7639ad78d668aa9aa675fed2"}
May 29 10:23:41.934852 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:41.934766    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerDied","Data":"f8f67f2cd40cf391a080db92592f71b07af9cca7f52c2759eba02fddcc0924e7"}
May 29 10:23:42.054722 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.054699    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:23:42.197544 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.197519    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/70a4f017-1b14-4308-a9cc-f2967b41b607-feast-data\") pod \"70a4f017-1b14-4308-a9cc-f2967b41b607\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") "
May 29 10:23:42.197673 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.197588    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-online-tls\") pod \"70a4f017-1b14-4308-a9cc-f2967b41b607\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") "
May 29 10:23:42.197727 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.197682    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxdql\" (UniqueName: \"kubernetes.io/projected/70a4f017-1b14-4308-a9cc-f2967b41b607-kube-api-access-dxdql\") pod \"70a4f017-1b14-4308-a9cc-f2967b41b607\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") "
May 29 10:23:42.197783 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.197741    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/70a4f017-1b14-4308-a9cc-f2967b41b607-registry-tls\") pod \"70a4f017-1b14-4308-a9cc-f2967b41b607\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") "
May 29 10:23:42.197783 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.197779    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-ui-tls\") pod \"70a4f017-1b14-4308-a9cc-f2967b41b607\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") "
May 29 10:23:42.197884 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.197802    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-offline-tls\") pod \"70a4f017-1b14-4308-a9cc-f2967b41b607\" (UID: \"70a4f017-1b14-4308-a9cc-f2967b41b607\") "
May 29 10:23:42.198186 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.198153    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a4f017-1b14-4308-a9cc-f2967b41b607-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "70a4f017-1b14-4308-a9cc-f2967b41b607" (UID: "70a4f017-1b14-4308-a9cc-f2967b41b607"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:23:42.198418 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.198356    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a4f017-1b14-4308-a9cc-f2967b41b607-feast-data" (OuterVolumeSpecName: "feast-data") pod "70a4f017-1b14-4308-a9cc-f2967b41b607" (UID: "70a4f017-1b14-4308-a9cc-f2967b41b607"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:23:42.200029 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.199998    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-online-tls" (OuterVolumeSpecName: "online-tls") pod "70a4f017-1b14-4308-a9cc-f2967b41b607" (UID: "70a4f017-1b14-4308-a9cc-f2967b41b607"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:23:42.200130 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.200107    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "70a4f017-1b14-4308-a9cc-f2967b41b607" (UID: "70a4f017-1b14-4308-a9cc-f2967b41b607"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:23:42.200279 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.200256    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a4f017-1b14-4308-a9cc-f2967b41b607-kube-api-access-dxdql" (OuterVolumeSpecName: "kube-api-access-dxdql") pod "70a4f017-1b14-4308-a9cc-f2967b41b607" (UID: "70a4f017-1b14-4308-a9cc-f2967b41b607"). InnerVolumeSpecName "kube-api-access-dxdql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:23:42.201256 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.201233    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "70a4f017-1b14-4308-a9cc-f2967b41b607" (UID: "70a4f017-1b14-4308-a9cc-f2967b41b607"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:23:42.202952 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.202935    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:23:42.299131 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299093    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-online-tls\") pod \"d27cc8ca-b289-4599-b149-538ad7b12e85\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") "
May 29 10:23:42.299315 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299150    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-registry-tls\") pod \"d27cc8ca-b289-4599-b149-538ad7b12e85\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") "
May 29 10:23:42.299315 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299172    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-offline-tls\") pod \"d27cc8ca-b289-4599-b149-538ad7b12e85\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") "
May 29 10:23:42.299315 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299196    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9zwg\" (UniqueName: \"kubernetes.io/projected/d27cc8ca-b289-4599-b149-538ad7b12e85-kube-api-access-v9zwg\") pod \"d27cc8ca-b289-4599-b149-538ad7b12e85\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") "
May 29 10:23:42.299315 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299212    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d27cc8ca-b289-4599-b149-538ad7b12e85-feast-data\") pod \"d27cc8ca-b289-4599-b149-538ad7b12e85\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") "
May 29 10:23:42.299315 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299232    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-ui-tls\") pod \"d27cc8ca-b289-4599-b149-538ad7b12e85\" (UID: \"d27cc8ca-b289-4599-b149-538ad7b12e85\") "
May 29 10:23:42.299667 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299491    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/70a4f017-1b14-4308-a9cc-f2967b41b607-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.299667 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299510    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.299667 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299526    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdql\" (UniqueName: \"kubernetes.io/projected/70a4f017-1b14-4308-a9cc-f2967b41b607-kube-api-access-dxdql\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.299667 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299541    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/70a4f017-1b14-4308-a9cc-f2967b41b607-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.299667 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299556    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-ui-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.299667 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299568    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/70a4f017-1b14-4308-a9cc-f2967b41b607-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.299927 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.299815    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27cc8ca-b289-4599-b149-538ad7b12e85-feast-data" (OuterVolumeSpecName: "feast-data") pod "d27cc8ca-b289-4599-b149-538ad7b12e85" (UID: "d27cc8ca-b289-4599-b149-538ad7b12e85"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:23:42.301427 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.301402    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27cc8ca-b289-4599-b149-538ad7b12e85-kube-api-access-v9zwg" (OuterVolumeSpecName: "kube-api-access-v9zwg") pod "d27cc8ca-b289-4599-b149-538ad7b12e85" (UID: "d27cc8ca-b289-4599-b149-538ad7b12e85"). InnerVolumeSpecName "kube-api-access-v9zwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:23:42.301890 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.301868    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-online-tls" (OuterVolumeSpecName: "online-tls") pod "d27cc8ca-b289-4599-b149-538ad7b12e85" (UID: "d27cc8ca-b289-4599-b149-538ad7b12e85"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:23:42.301961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.301887    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d27cc8ca-b289-4599-b149-538ad7b12e85" (UID: "d27cc8ca-b289-4599-b149-538ad7b12e85"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:23:42.301961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.301907    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d27cc8ca-b289-4599-b149-538ad7b12e85" (UID: "d27cc8ca-b289-4599-b149-538ad7b12e85"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:23:42.301961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.301916    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d27cc8ca-b289-4599-b149-538ad7b12e85" (UID: "d27cc8ca-b289-4599-b149-538ad7b12e85"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:23:42.400179 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.400144    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.400179 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.400173    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.400179 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.400182    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.400179 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.400190    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9zwg\" (UniqueName: \"kubernetes.io/projected/d27cc8ca-b289-4599-b149-538ad7b12e85-kube-api-access-v9zwg\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.400437 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.400199    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d27cc8ca-b289-4599-b149-538ad7b12e85-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.400437 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.400209    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d27cc8ca-b289-4599-b149-538ad7b12e85-ui-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:23:42.940608 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.940571    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9" event={"ID":"70a4f017-1b14-4308-a9cc-f2967b41b607","Type":"ContainerDied","Data":"76c6fbe82fa940ac567c0eb5753222fefce772fe88ee48d04bd8cc71bcfc04d3"}
May 29 10:23:42.940608 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.940594    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"
May 29 10:23:42.941156 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.940617    2562 scope.go:117] "RemoveContainer" containerID="83aad4186d86107984548ba316145eedd6200d5087fde3815b3635824d7f70bb"
May 29 10:23:42.943476 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.943449    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8" event={"ID":"d27cc8ca-b289-4599-b149-538ad7b12e85","Type":"ContainerDied","Data":"655a7554f63bdc450dd21c786936651ec27d79677a28c9a5aaf302f31e60a745"}
May 29 10:23:42.943558 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.943485    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"
May 29 10:23:42.950577 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.950563    2562 scope.go:117] "RemoveContainer" containerID="944a13800627a89c8ba2449c85625782a5ce46643d9e9674ef2b4a7668e44bd9"
May 29 10:23:42.958419 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.958403    2562 scope.go:117] "RemoveContainer" containerID="4dc4f90b04561b5d2128f1ab5a975330fe3c91be0f69656ffe09d03caf2c9f09"
May 29 10:23:42.963674 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.963590    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"]
May 29 10:23:42.966763 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.966748    2562 scope.go:117] "RemoveContainer" containerID="e1a8e60c5ff3c91d3eea3470fb39b2585bc6b5461f74c61f0e4aaac7c0f718e0"
May 29 10:23:42.969712 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.969689    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c9df695d5-r24k9"]
May 29 10:23:42.974488 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.974474    2562 scope.go:117] "RemoveContainer" containerID="42a9b01a4397020b700be782f67952f9b073e54536989caf16a678cc62956844"
May 29 10:23:42.981137 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.981116    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"]
May 29 10:23:42.982622 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.982606    2562 scope.go:117] "RemoveContainer" containerID="c438738e7fdd69cf88f1b26ac94308c461bcf1bd8761f8ba455bbbc9b0eccb4b"
May 29 10:23:42.987579 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.987556    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-754cfff778-wxng8"]
May 29 10:23:42.990955 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.990938    2562 scope.go:117] "RemoveContainer" containerID="d0a85e27518b1467b4ea5c826184db3bd3e5603e7639ad78d668aa9aa675fed2"
May 29 10:23:42.998201 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:42.998187    2562 scope.go:117] "RemoveContainer" containerID="12d20bedb94a2c9c85d70f4682bb460a218c36c00654c6fc1997847f3f55eb1a"
May 29 10:23:43.005289 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:43.005271    2562 scope.go:117] "RemoveContainer" containerID="f8f67f2cd40cf391a080db92592f71b07af9cca7f52c2759eba02fddcc0924e7"
May 29 10:23:43.012468 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:43.012452    2562 scope.go:117] "RemoveContainer" containerID="17bf4396668b1fcc1f86792bb4ea1623d084433b49c04c26bbcba3589a161008"
May 29 10:23:43.019566 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:43.019551    2562 scope.go:117] "RemoveContainer" containerID="a9d5a283c81296feb8e4c021de66aff728c9d1c0ee26a825ad0810adfd00a0c1"
May 29 10:23:43.703482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:43.703445    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" path="/var/lib/kubelet/pods/70a4f017-1b14-4308-a9cc-f2967b41b607/volumes"
May 29 10:23:43.704035 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:43.704020    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" path="/var/lib/kubelet/pods/d27cc8ca-b289-4599-b149-538ad7b12e85/volumes"
May 29 10:23:44.969616 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969582    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-whn69"]
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969932    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="feast-apply"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969944    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="feast-apply"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969951    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="online"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969957    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="online"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969966    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="online"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969971    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="online"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969978    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="feast-apply"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969983    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="feast-apply"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969988    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="ui"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.969993    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="ui"
May 29 10:23:44.969993 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970000    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="offline"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970005    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="offline"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970011    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="feast-init"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970016    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="feast-init"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970026    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="ui"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970030    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="ui"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970037    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="offline"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970043    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="offline"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970051    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="feast-init"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970056    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="feast-init"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970062    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="registry"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970066    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="registry"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970111    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="registry"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970121    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="offline"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970127    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="ui"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970133    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="online"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970140    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="ui"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970145    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="d27cc8ca-b289-4599-b149-538ad7b12e85" containerName="offline"
May 29 10:23:44.970345 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.970151    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="70a4f017-1b14-4308-a9cc-f2967b41b607" containerName="online"
May 29 10:23:44.974081 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.974062    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:23:44.978453 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.978430    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 10:23:44.978580 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.978475    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 10:23:44.978950 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.978930    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-j24fz\""
May 29 10:23:44.980984 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.980912    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 29 10:23:44.989479 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:44.989457    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-whn69"]
May 29 10:23:45.042445 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.042405    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-854544c579-xwfp7"]
May 29 10:23:45.045537 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.045520    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-xwfp7"
May 29 10:23:45.054429 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.054402    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-xwfp7"]
May 29 10:23:45.125533 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.125490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/04091ed8-5aeb-401e-b152-7a2933cb03aa-postgresdata\") pod \"postgres-7c7656cc6d-whn69\" (UID: \"04091ed8-5aeb-401e-b152-7a2933cb03aa\") " pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:23:45.125533 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.125537    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k5jh\" (UniqueName: \"kubernetes.io/projected/b9d16894-3a4d-4ea1-87ce-dc9821a05d85-kube-api-access-2k5jh\") pod \"redis-854544c579-xwfp7\" (UID: \"b9d16894-3a4d-4ea1-87ce-dc9821a05d85\") " pod="test-ns-feast/redis-854544c579-xwfp7"
May 29 10:23:45.125763 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.125615    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4kz\" (UniqueName: \"kubernetes.io/projected/04091ed8-5aeb-401e-b152-7a2933cb03aa-kube-api-access-zg4kz\") pod \"postgres-7c7656cc6d-whn69\" (UID: \"04091ed8-5aeb-401e-b152-7a2933cb03aa\") " pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:23:45.227087 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.226990    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/04091ed8-5aeb-401e-b152-7a2933cb03aa-postgresdata\") pod \"postgres-7c7656cc6d-whn69\" (UID: \"04091ed8-5aeb-401e-b152-7a2933cb03aa\") " pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:23:45.227087 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.227050    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5jh\" (UniqueName: \"kubernetes.io/projected/b9d16894-3a4d-4ea1-87ce-dc9821a05d85-kube-api-access-2k5jh\") pod \"redis-854544c579-xwfp7\" (UID: \"b9d16894-3a4d-4ea1-87ce-dc9821a05d85\") " pod="test-ns-feast/redis-854544c579-xwfp7"
May 29 10:23:45.227087 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.227081    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4kz\" (UniqueName: \"kubernetes.io/projected/04091ed8-5aeb-401e-b152-7a2933cb03aa-kube-api-access-zg4kz\") pod \"postgres-7c7656cc6d-whn69\" (UID: \"04091ed8-5aeb-401e-b152-7a2933cb03aa\") " pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:23:45.227399 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.227377    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/04091ed8-5aeb-401e-b152-7a2933cb03aa-postgresdata\") pod \"postgres-7c7656cc6d-whn69\" (UID: \"04091ed8-5aeb-401e-b152-7a2933cb03aa\") " pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:23:45.235440 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.235415    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4kz\" (UniqueName: \"kubernetes.io/projected/04091ed8-5aeb-401e-b152-7a2933cb03aa-kube-api-access-zg4kz\") pod \"postgres-7c7656cc6d-whn69\" (UID: \"04091ed8-5aeb-401e-b152-7a2933cb03aa\") " pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:23:45.235771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.235751    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5jh\" (UniqueName: \"kubernetes.io/projected/b9d16894-3a4d-4ea1-87ce-dc9821a05d85-kube-api-access-2k5jh\") pod \"redis-854544c579-xwfp7\" (UID: \"b9d16894-3a4d-4ea1-87ce-dc9821a05d85\") " pod="test-ns-feast/redis-854544c579-xwfp7"
May 29 10:23:45.293599 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.293550    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:23:45.356810 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.356601    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-xwfp7"
May 29 10:23:45.432162 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.432124    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-whn69"]
May 29 10:23:45.434662 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:23:45.434613    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04091ed8_5aeb_401e_b152_7a2933cb03aa.slice/crio-6c5f0543c2cbc7daf77db4bab106348e7b598e2b7f02798d022905caac815015 WatchSource:0}: Error finding container 6c5f0543c2cbc7daf77db4bab106348e7b598e2b7f02798d022905caac815015: Status 404 returned error can't find the container with id 6c5f0543c2cbc7daf77db4bab106348e7b598e2b7f02798d022905caac815015
May 29 10:23:45.436447 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.436428    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 10:23:45.491774 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.491571    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-xwfp7"]
May 29 10:23:45.494334 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:23:45.494307    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9d16894_3a4d_4ea1_87ce_dc9821a05d85.slice/crio-7189c8880fc65d4601f9dc783546e10f298b0a6f97df5c9a27aff759009c9937 WatchSource:0}: Error finding container 7189c8880fc65d4601f9dc783546e10f298b0a6f97df5c9a27aff759009c9937: Status 404 returned error can't find the container with id 7189c8880fc65d4601f9dc783546e10f298b0a6f97df5c9a27aff759009c9937
May 29 10:23:45.954203 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.954164    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-whn69" event={"ID":"04091ed8-5aeb-401e-b152-7a2933cb03aa","Type":"ContainerStarted","Data":"6c5f0543c2cbc7daf77db4bab106348e7b598e2b7f02798d022905caac815015"}
May 29 10:23:45.955141 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:45.955119    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-xwfp7" event={"ID":"b9d16894-3a4d-4ea1-87ce-dc9821a05d85","Type":"ContainerStarted","Data":"7189c8880fc65d4601f9dc783546e10f298b0a6f97df5c9a27aff759009c9937"}
May 29 10:23:47.963523 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:47.963485    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-xwfp7" event={"ID":"b9d16894-3a4d-4ea1-87ce-dc9821a05d85","Type":"ContainerStarted","Data":"2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225"}
May 29 10:23:47.979392 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:47.979343    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/redis-854544c579-xwfp7" podStartSLOduration=1.36947545 podStartE2EDuration="2.979328409s" podCreationTimestamp="2026-05-29 10:23:45 +0000 UTC" firstStartedPulling="2026-05-29 10:23:45.496116617 +0000 UTC m=+648.392676214" lastFinishedPulling="2026-05-29 10:23:47.105969567 +0000 UTC m=+650.002529173" observedRunningTime="2026-05-29 10:23:47.977221865 +0000 UTC m=+650.873781484" watchObservedRunningTime="2026-05-29 10:23:47.979328409 +0000 UTC m=+650.875888028"
May 29 10:23:51.982129 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:51.982086    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-whn69" event={"ID":"04091ed8-5aeb-401e-b152-7a2933cb03aa","Type":"ContainerStarted","Data":"48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7"}
May 29 10:23:51.997608 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:51.997560    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-whn69" podStartSLOduration=2.323724828 podStartE2EDuration="7.997546654s" podCreationTimestamp="2026-05-29 10:23:44 +0000 UTC" firstStartedPulling="2026-05-29 10:23:45.436552616 +0000 UTC m=+648.333112215" lastFinishedPulling="2026-05-29 10:23:51.110374442 +0000 UTC m=+654.006934041" observedRunningTime="2026-05-29 10:23:51.996344933 +0000 UTC m=+654.892904553" watchObservedRunningTime="2026-05-29 10:23:51.997546654 +0000 UTC m=+654.894106368"
May 29 10:23:55.756434 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.756397    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"]
May 29 10:23:55.763118 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.763094    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:55.765339 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.765320    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 29 10:23:55.765492 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.765469    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 29 10:23:55.765600 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.765552    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 29 10:23:55.765600 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.765559    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-l4224\""
May 29 10:23:55.765838 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.765824    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 29 10:23:55.770621 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.770599    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"]
May 29 10:23:55.924087 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.924055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzvf\" (UniqueName: \"kubernetes.io/projected/971b7c53-47c3-4e78-90c1-a655a4d39bd7-kube-api-access-5dzvf\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:55.924261 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.924095    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-offline-tls\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:55.924261 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.924124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-registry-tls\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:55.924261 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.924211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/971b7c53-47c3-4e78-90c1-a655a4d39bd7-feast-data\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:55.924372 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:55.924310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-online-tls\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.024896 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.024812    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-online-tls\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.024896 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.024868    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzvf\" (UniqueName: \"kubernetes.io/projected/971b7c53-47c3-4e78-90c1-a655a4d39bd7-kube-api-access-5dzvf\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.025086 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.024899    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-offline-tls\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.025086 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.024937    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-registry-tls\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.025086 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.024976    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/971b7c53-47c3-4e78-90c1-a655a4d39bd7-feast-data\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.025360 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.025341    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/971b7c53-47c3-4e78-90c1-a655a4d39bd7-feast-data\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.027563 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.027530    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-offline-tls\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.027563 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.027559    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-online-tls\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.027731 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.027587    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-registry-tls\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.033066 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.033042    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzvf\" (UniqueName: \"kubernetes.io/projected/971b7c53-47c3-4e78-90c1-a655a4d39bd7-kube-api-access-5dzvf\") pod \"feast-credit-scoring-6b74c65946-r4qch\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") " pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.074170 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.074134    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:23:56.213431 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.213407    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"]
May 29 10:23:56.215272 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:23:56.215238    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971b7c53_47c3_4e78_90c1_a655a4d39bd7.slice/crio-92dc9f90c05563627efdf383b194a1530666b09ad08690a34cf7347ebc40f020 WatchSource:0}: Error finding container 92dc9f90c05563627efdf383b194a1530666b09ad08690a34cf7347ebc40f020: Status 404 returned error can't find the container with id 92dc9f90c05563627efdf383b194a1530666b09ad08690a34cf7347ebc40f020
May 29 10:23:56.998740 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.998706    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerStarted","Data":"9ef88d7fe2f7318397543c02e3e332f882d84f71c3331ea3342b671981e27d55"}
May 29 10:23:56.998740 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:56.998744    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerStarted","Data":"92dc9f90c05563627efdf383b194a1530666b09ad08690a34cf7347ebc40f020"}
May 29 10:23:58.002693 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:58.002606    2562 generic.go:358] "Generic (PLEG): container finished" podID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerID="9ef88d7fe2f7318397543c02e3e332f882d84f71c3331ea3342b671981e27d55" exitCode=0
May 29 10:23:58.002693 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:58.002680    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerDied","Data":"9ef88d7fe2f7318397543c02e3e332f882d84f71c3331ea3342b671981e27d55"}
May 29 10:23:59.007044 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:23:59.007010    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerStarted","Data":"a6e5f5192463f158a6c1bde5748bf0a2ba223e29350e5852109cf14091c9cb3b"}
May 29 10:24:05.033529 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:05.033492    2562 generic.go:358] "Generic (PLEG): container finished" podID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerID="a6e5f5192463f158a6c1bde5748bf0a2ba223e29350e5852109cf14091c9cb3b" exitCode=0
May 29 10:24:05.033989 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:05.033567    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerDied","Data":"a6e5f5192463f158a6c1bde5748bf0a2ba223e29350e5852109cf14091c9cb3b"}
May 29 10:24:05.036792 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:05.036772    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 29 10:24:06.040103 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:06.040065    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerStarted","Data":"4b26dc29661a8535980fc5a66e742867790cfa24ec7ba34d0d72dca321b3c292"}
May 29 10:24:06.040103 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:06.040105    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerStarted","Data":"5b0a05a2ea341643978be40ef2d27a930ee9680bc09f8519142bc56252b525ed"}
May 29 10:24:06.040656 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:06.040116    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerStarted","Data":"f72b90626a0c5f2d0983e03bc658e5fcf34b0efdef5da494aae138f13b360feb"}
May 29 10:24:06.068834 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:06.068763    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podStartSLOduration=11.068743527 podStartE2EDuration="11.068743527s" podCreationTimestamp="2026-05-29 10:23:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:24:06.066273246 +0000 UTC m=+668.962832866" watchObservedRunningTime="2026-05-29 10:24:06.068743527 +0000 UTC m=+668.965303145"
May 29 10:24:06.075039 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:06.075008    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:06.075039 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:06.075042    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:06.075262 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:06.075051    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:08.075281 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:08.075245    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:08.075865 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:08.075835    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:08.075865 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:08.075863    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:08.076729 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:08.076584    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused"
May 29 10:24:08.076729 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:08.076608    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused"
May 29 10:24:08.076729 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:08.076661    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 29 10:24:11.074758 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:11.074705    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused"
May 29 10:24:11.075773 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:11.075748    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:11.076820 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:11.076794    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:11.206403 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:11.206370    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:12.065194 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:12.065162    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:14.075482 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:14.075436    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused"
May 29 10:24:17.074974 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:17.074942    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:24:17.075817 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:24:17.075802    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:25:18.804372 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:18.804332    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"]
May 29 10:25:18.804951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:18.804695    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="registry" containerID="cri-o://f72b90626a0c5f2d0983e03bc658e5fcf34b0efdef5da494aae138f13b360feb" gracePeriod=30
May 29 10:25:18.804951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:18.804757    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="offline" containerID="cri-o://4b26dc29661a8535980fc5a66e742867790cfa24ec7ba34d0d72dca321b3c292" gracePeriod=30
May 29 10:25:18.804951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:18.804755    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="online" containerID="cri-o://5b0a05a2ea341643978be40ef2d27a930ee9680bc09f8519142bc56252b525ed" gracePeriod=30
May 29 10:25:18.826730 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:18.826697    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-whn69"]
May 29 10:25:18.827188 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:18.827152    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-whn69" podUID="04091ed8-5aeb-401e-b152-7a2933cb03aa" containerName="postgres" containerID="cri-o://48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7" gracePeriod=30
May 29 10:25:18.836372 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:18.836342    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-xwfp7"]
May 29 10:25:18.836580 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:18.836541    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/redis-854544c579-xwfp7" podUID="b9d16894-3a4d-4ea1-87ce-dc9821a05d85" containerName="redis" containerID="cri-o://2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225" gracePeriod=30
May 29 10:25:19.094051 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.094026    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:25:19.097456 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.097429    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-xwfp7"
May 29 10:25:19.272682 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.272628    2562 generic.go:358] "Generic (PLEG): container finished" podID="04091ed8-5aeb-401e-b152-7a2933cb03aa" containerID="48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7" exitCode=0
May 29 10:25:19.272879 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.272703    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-whn69"
May 29 10:25:19.272879 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.272733    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-whn69" event={"ID":"04091ed8-5aeb-401e-b152-7a2933cb03aa","Type":"ContainerDied","Data":"48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7"}
May 29 10:25:19.272879 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.272779    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-whn69" event={"ID":"04091ed8-5aeb-401e-b152-7a2933cb03aa","Type":"ContainerDied","Data":"6c5f0543c2cbc7daf77db4bab106348e7b598e2b7f02798d022905caac815015"}
May 29 10:25:19.272879 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.272804    2562 scope.go:117] "RemoveContainer" containerID="48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7"
May 29 10:25:19.274063 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.274039    2562 generic.go:358] "Generic (PLEG): container finished" podID="b9d16894-3a4d-4ea1-87ce-dc9821a05d85" containerID="2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225" exitCode=0
May 29 10:25:19.274168 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.274082    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-xwfp7" event={"ID":"b9d16894-3a4d-4ea1-87ce-dc9821a05d85","Type":"ContainerDied","Data":"2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225"}
May 29 10:25:19.274168 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.274100    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-xwfp7"
May 29 10:25:19.274168 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.274109    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-xwfp7" event={"ID":"b9d16894-3a4d-4ea1-87ce-dc9821a05d85","Type":"ContainerDied","Data":"7189c8880fc65d4601f9dc783546e10f298b0a6f97df5c9a27aff759009c9937"}
May 29 10:25:19.278968 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.278942    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg4kz\" (UniqueName: \"kubernetes.io/projected/04091ed8-5aeb-401e-b152-7a2933cb03aa-kube-api-access-zg4kz\") pod \"04091ed8-5aeb-401e-b152-7a2933cb03aa\" (UID: \"04091ed8-5aeb-401e-b152-7a2933cb03aa\") "
May 29 10:25:19.279062 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.279003    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k5jh\" (UniqueName: \"kubernetes.io/projected/b9d16894-3a4d-4ea1-87ce-dc9821a05d85-kube-api-access-2k5jh\") pod \"b9d16894-3a4d-4ea1-87ce-dc9821a05d85\" (UID: \"b9d16894-3a4d-4ea1-87ce-dc9821a05d85\") "
May 29 10:25:19.279138 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.279118    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/04091ed8-5aeb-401e-b152-7a2933cb03aa-postgresdata\") pod \"04091ed8-5aeb-401e-b152-7a2933cb03aa\" (UID: \"04091ed8-5aeb-401e-b152-7a2933cb03aa\") "
May 29 10:25:19.282742 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.282709    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d16894-3a4d-4ea1-87ce-dc9821a05d85-kube-api-access-2k5jh" (OuterVolumeSpecName: "kube-api-access-2k5jh") pod "b9d16894-3a4d-4ea1-87ce-dc9821a05d85" (UID: "b9d16894-3a4d-4ea1-87ce-dc9821a05d85"). InnerVolumeSpecName "kube-api-access-2k5jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:25:19.284262 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.284218    2562 scope.go:117] "RemoveContainer" containerID="48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7"
May 29 10:25:19.284381 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.284214    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04091ed8-5aeb-401e-b152-7a2933cb03aa-kube-api-access-zg4kz" (OuterVolumeSpecName: "kube-api-access-zg4kz") pod "04091ed8-5aeb-401e-b152-7a2933cb03aa" (UID: "04091ed8-5aeb-401e-b152-7a2933cb03aa"). InnerVolumeSpecName "kube-api-access-zg4kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:25:19.284839 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:25:19.284817    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7\": container with ID starting with 48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7 not found: ID does not exist" containerID="48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7"
May 29 10:25:19.284947 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.284851    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7"} err="failed to get container status \"48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7\": rpc error: code = NotFound desc = could not find container \"48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7\": container with ID starting with 48586474d417fe125a5d1f82c395a83fffce8c48e670ea505d9fb6be6edb46f7 not found: ID does not exist"
May 29 10:25:19.284947 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.284877    2562 scope.go:117] "RemoveContainer" containerID="2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225"
May 29 10:25:19.293453 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.293438    2562 scope.go:117] "RemoveContainer" containerID="2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225"
May 29 10:25:19.293785 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:25:19.293764    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225\": container with ID starting with 2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225 not found: ID does not exist" containerID="2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225"
May 29 10:25:19.293868 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.293791    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225"} err="failed to get container status \"2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225\": rpc error: code = NotFound desc = could not find container \"2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225\": container with ID starting with 2aaf8950ec8ffce203507b7192e5627cd1f8024a4c278d431100153dd0c07225 not found: ID does not exist"
May 29 10:25:19.315697 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.315657    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04091ed8-5aeb-401e-b152-7a2933cb03aa-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "04091ed8-5aeb-401e-b152-7a2933cb03aa" (UID: "04091ed8-5aeb-401e-b152-7a2933cb03aa"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:25:19.380464 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.380423    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg4kz\" (UniqueName: \"kubernetes.io/projected/04091ed8-5aeb-401e-b152-7a2933cb03aa-kube-api-access-zg4kz\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:25:19.380464 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.380461    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k5jh\" (UniqueName: \"kubernetes.io/projected/b9d16894-3a4d-4ea1-87ce-dc9821a05d85-kube-api-access-2k5jh\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:25:19.380677 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.380475    2562 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/04091ed8-5aeb-401e-b152-7a2933cb03aa-postgresdata\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:25:19.597876 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.597843    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-xwfp7"]
May 29 10:25:19.600747 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.600709    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-854544c579-xwfp7"]
May 29 10:25:19.610275 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.610237    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-whn69"]
May 29 10:25:19.614992 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.614966    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-whn69"]
May 29 10:25:19.703683 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.703580    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04091ed8-5aeb-401e-b152-7a2933cb03aa" path="/var/lib/kubelet/pods/04091ed8-5aeb-401e-b152-7a2933cb03aa/volumes"
May 29 10:25:19.703969 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:19.703955    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d16894-3a4d-4ea1-87ce-dc9821a05d85" path="/var/lib/kubelet/pods/b9d16894-3a4d-4ea1-87ce-dc9821a05d85/volumes"
May 29 10:25:20.279272 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:20.279239    2562 generic.go:358] "Generic (PLEG): container finished" podID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerID="5b0a05a2ea341643978be40ef2d27a930ee9680bc09f8519142bc56252b525ed" exitCode=0
May 29 10:25:20.279706 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:20.279318    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerDied","Data":"5b0a05a2ea341643978be40ef2d27a930ee9680bc09f8519142bc56252b525ed"}
May 29 10:25:22.062847 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:22.062802    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 29 10:25:32.062609 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:32.062563    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 29 10:25:42.062315 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:42.062274    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 29 10:25:42.062703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:42.062399    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:25:49.368231 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.368198    2562 generic.go:358] "Generic (PLEG): container finished" podID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerID="4b26dc29661a8535980fc5a66e742867790cfa24ec7ba34d0d72dca321b3c292" exitCode=137
May 29 10:25:49.368231 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.368226    2562 generic.go:358] "Generic (PLEG): container finished" podID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerID="f72b90626a0c5f2d0983e03bc658e5fcf34b0efdef5da494aae138f13b360feb" exitCode=137
May 29 10:25:49.368605 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.368263    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerDied","Data":"4b26dc29661a8535980fc5a66e742867790cfa24ec7ba34d0d72dca321b3c292"}
May 29 10:25:49.368605 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.368293    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerDied","Data":"f72b90626a0c5f2d0983e03bc658e5fcf34b0efdef5da494aae138f13b360feb"}
May 29 10:25:49.450107 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.450082    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:25:49.532220 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.532188    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-registry-tls\") pod \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") "
May 29 10:25:49.532368 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.532247    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-offline-tls\") pod \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") "
May 29 10:25:49.532368 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.532275    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/971b7c53-47c3-4e78-90c1-a655a4d39bd7-feast-data\") pod \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") "
May 29 10:25:49.532368 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.532301    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dzvf\" (UniqueName: \"kubernetes.io/projected/971b7c53-47c3-4e78-90c1-a655a4d39bd7-kube-api-access-5dzvf\") pod \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") "
May 29 10:25:49.532368 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.532334    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-online-tls\") pod \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\" (UID: \"971b7c53-47c3-4e78-90c1-a655a4d39bd7\") "
May 29 10:25:49.534528 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.534484    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971b7c53-47c3-4e78-90c1-a655a4d39bd7-kube-api-access-5dzvf" (OuterVolumeSpecName: "kube-api-access-5dzvf") pod "971b7c53-47c3-4e78-90c1-a655a4d39bd7" (UID: "971b7c53-47c3-4e78-90c1-a655a4d39bd7"). InnerVolumeSpecName "kube-api-access-5dzvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:25:49.534687 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.534580    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "971b7c53-47c3-4e78-90c1-a655a4d39bd7" (UID: "971b7c53-47c3-4e78-90c1-a655a4d39bd7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:25:49.534824 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.534794    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "971b7c53-47c3-4e78-90c1-a655a4d39bd7" (UID: "971b7c53-47c3-4e78-90c1-a655a4d39bd7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:25:49.535109 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.535088    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-online-tls" (OuterVolumeSpecName: "online-tls") pod "971b7c53-47c3-4e78-90c1-a655a4d39bd7" (UID: "971b7c53-47c3-4e78-90c1-a655a4d39bd7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:25:49.540775 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.540753    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971b7c53-47c3-4e78-90c1-a655a4d39bd7-feast-data" (OuterVolumeSpecName: "feast-data") pod "971b7c53-47c3-4e78-90c1-a655a4d39bd7" (UID: "971b7c53-47c3-4e78-90c1-a655a4d39bd7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:25:49.633506 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.633473    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dzvf\" (UniqueName: \"kubernetes.io/projected/971b7c53-47c3-4e78-90c1-a655a4d39bd7-kube-api-access-5dzvf\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:25:49.633506 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.633505    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:25:49.633744 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.633515    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:25:49.633744 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.633524    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/971b7c53-47c3-4e78-90c1-a655a4d39bd7-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:25:49.633744 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:49.633533    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/971b7c53-47c3-4e78-90c1-a655a4d39bd7-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:25:50.373743 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:50.373702    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch" event={"ID":"971b7c53-47c3-4e78-90c1-a655a4d39bd7","Type":"ContainerDied","Data":"92dc9f90c05563627efdf383b194a1530666b09ad08690a34cf7347ebc40f020"}
May 29 10:25:50.373743 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:50.373751    2562 scope.go:117] "RemoveContainer" containerID="4b26dc29661a8535980fc5a66e742867790cfa24ec7ba34d0d72dca321b3c292"
May 29 10:25:50.374304 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:50.373813    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"
May 29 10:25:50.381882 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:50.381809    2562 scope.go:117] "RemoveContainer" containerID="5b0a05a2ea341643978be40ef2d27a930ee9680bc09f8519142bc56252b525ed"
May 29 10:25:50.388931 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:50.388915    2562 scope.go:117] "RemoveContainer" containerID="f72b90626a0c5f2d0983e03bc658e5fcf34b0efdef5da494aae138f13b360feb"
May 29 10:25:50.396504 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:50.396489    2562 scope.go:117] "RemoveContainer" containerID="a6e5f5192463f158a6c1bde5748bf0a2ba223e29350e5852109cf14091c9cb3b"
May 29 10:25:50.397413 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:50.397337    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"]
May 29 10:25:50.401654 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:50.401619    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b74c65946-r4qch"]
May 29 10:25:50.404174 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:50.404153    2562 scope.go:117] "RemoveContainer" containerID="9ef88d7fe2f7318397543c02e3e332f882d84f71c3331ea3342b671981e27d55"
May 29 10:25:51.703100 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:25:51.703066    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" path="/var/lib/kubelet/pods/971b7c53-47c3-4e78-90c1-a655a4d39bd7/volumes"
May 29 10:27:35.580867 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.580833    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-5kbnn"]
May 29 10:27:35.581430 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581353    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d16894-3a4d-4ea1-87ce-dc9821a05d85" containerName="redis"
May 29 10:27:35.581430 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581375    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d16894-3a4d-4ea1-87ce-dc9821a05d85" containerName="redis"
May 29 10:27:35.581430 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581388    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="offline"
May 29 10:27:35.581430 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581398    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="offline"
May 29 10:27:35.581430 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581415    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="feast-apply"
May 29 10:27:35.581430 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581425    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="feast-apply"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581443    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="online"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581452    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="online"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581464    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="feast-init"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581471    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="feast-init"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581484    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04091ed8-5aeb-401e-b152-7a2933cb03aa" containerName="postgres"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581492    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04091ed8-5aeb-401e-b152-7a2933cb03aa" containerName="postgres"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581509    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="registry"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581517    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="registry"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581594    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b9d16894-3a4d-4ea1-87ce-dc9821a05d85" containerName="redis"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581606    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="04091ed8-5aeb-401e-b152-7a2933cb03aa" containerName="postgres"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581616    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="registry"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581629    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="offline"
May 29 10:27:35.581782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.581670    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="971b7c53-47c3-4e78-90c1-a655a4d39bd7" containerName="online"
May 29 10:27:35.584900 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.584880    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-5kbnn"
May 29 10:27:35.587042 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.587018    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-ff242\""
May 29 10:27:35.587161 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.587051    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 29 10:27:35.587347 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.587329    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 29 10:27:35.599519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.599489    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-5kbnn"]
May 29 10:27:35.716377 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.716349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6mrh\" (UniqueName: \"kubernetes.io/projected/4d7d606c-c643-4eb0-9df5-a38f2da4bfe5-kube-api-access-l6mrh\") pod \"redis-854544c579-5kbnn\" (UID: \"4d7d606c-c643-4eb0-9df5-a38f2da4bfe5\") " pod="test-ns-feast-rest/redis-854544c579-5kbnn"
May 29 10:27:35.817275 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.817244    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l6mrh\" (UniqueName: \"kubernetes.io/projected/4d7d606c-c643-4eb0-9df5-a38f2da4bfe5-kube-api-access-l6mrh\") pod \"redis-854544c579-5kbnn\" (UID: \"4d7d606c-c643-4eb0-9df5-a38f2da4bfe5\") " pod="test-ns-feast-rest/redis-854544c579-5kbnn"
May 29 10:27:35.826714 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.826693    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6mrh\" (UniqueName: \"kubernetes.io/projected/4d7d606c-c643-4eb0-9df5-a38f2da4bfe5-kube-api-access-l6mrh\") pod \"redis-854544c579-5kbnn\" (UID: \"4d7d606c-c643-4eb0-9df5-a38f2da4bfe5\") " pod="test-ns-feast-rest/redis-854544c579-5kbnn"
May 29 10:27:35.934809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:35.934764    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-5kbnn"
May 29 10:27:36.062125 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:36.062042    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-5kbnn"]
May 29 10:27:36.064360 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:27:36.064330    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d7d606c_c643_4eb0_9df5_a38f2da4bfe5.slice/crio-bbb422bd598f27e413553eea5d267e1460f24926db011f3ef502772ea325a495 WatchSource:0}: Error finding container bbb422bd598f27e413553eea5d267e1460f24926db011f3ef502772ea325a495: Status 404 returned error can't find the container with id bbb422bd598f27e413553eea5d267e1460f24926db011f3ef502772ea325a495
May 29 10:27:36.698793 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:36.698752    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-5kbnn" event={"ID":"4d7d606c-c643-4eb0-9df5-a38f2da4bfe5","Type":"ContainerStarted","Data":"c7676f26eade41b4c1c482280c3ba7ddb835a4f5e90d43dcd2b48faa56ea8ed2"}
May 29 10:27:36.699195 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:36.698796    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-5kbnn" event={"ID":"4d7d606c-c643-4eb0-9df5-a38f2da4bfe5","Type":"ContainerStarted","Data":"bbb422bd598f27e413553eea5d267e1460f24926db011f3ef502772ea325a495"}
May 29 10:27:36.713350 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:36.713305    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-5kbnn" podStartSLOduration=1.7132931569999998 podStartE2EDuration="1.713293157s" podCreationTimestamp="2026-05-29 10:27:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:27:36.712127935 +0000 UTC m=+879.608687553" watchObservedRunningTime="2026-05-29 10:27:36.713293157 +0000 UTC m=+879.609852818"
May 29 10:27:41.014186 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.014098    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-ps94l"]
May 29 10:27:41.017593 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.017569    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:27:41.019404 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.019383    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 29 10:27:41.025429 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.025400    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-ps94l"]
May 29 10:27:41.165549 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.165512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/05c7df0a-f480-4f7f-8801-53c8e08172e3-postgresdata\") pod \"postgres-7c7656cc6d-ps94l\" (UID: \"05c7df0a-f480-4f7f-8801-53c8e08172e3\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:27:41.165759 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.165588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq7jb\" (UniqueName: \"kubernetes.io/projected/05c7df0a-f480-4f7f-8801-53c8e08172e3-kube-api-access-bq7jb\") pod \"postgres-7c7656cc6d-ps94l\" (UID: \"05c7df0a-f480-4f7f-8801-53c8e08172e3\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:27:41.267097 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.267014    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bq7jb\" (UniqueName: \"kubernetes.io/projected/05c7df0a-f480-4f7f-8801-53c8e08172e3-kube-api-access-bq7jb\") pod \"postgres-7c7656cc6d-ps94l\" (UID: \"05c7df0a-f480-4f7f-8801-53c8e08172e3\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:27:41.267097 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.267070    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/05c7df0a-f480-4f7f-8801-53c8e08172e3-postgresdata\") pod \"postgres-7c7656cc6d-ps94l\" (UID: \"05c7df0a-f480-4f7f-8801-53c8e08172e3\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:27:41.267371 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.267347    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/05c7df0a-f480-4f7f-8801-53c8e08172e3-postgresdata\") pod \"postgres-7c7656cc6d-ps94l\" (UID: \"05c7df0a-f480-4f7f-8801-53c8e08172e3\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:27:41.275725 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.275700    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq7jb\" (UniqueName: \"kubernetes.io/projected/05c7df0a-f480-4f7f-8801-53c8e08172e3-kube-api-access-bq7jb\") pod \"postgres-7c7656cc6d-ps94l\" (UID: \"05c7df0a-f480-4f7f-8801-53c8e08172e3\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:27:41.329666 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.329613    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:27:41.454355 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.454185    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-ps94l"]
May 29 10:27:41.457091 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:27:41.457063    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05c7df0a_f480_4f7f_8801_53c8e08172e3.slice/crio-4acb8bdabe24ab4026bf5c6c9f21f394f033677057408eeec9dd85f556e3307a WatchSource:0}: Error finding container 4acb8bdabe24ab4026bf5c6c9f21f394f033677057408eeec9dd85f556e3307a: Status 404 returned error can't find the container with id 4acb8bdabe24ab4026bf5c6c9f21f394f033677057408eeec9dd85f556e3307a
May 29 10:27:41.715337 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.715297    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l" event={"ID":"05c7df0a-f480-4f7f-8801-53c8e08172e3","Type":"ContainerStarted","Data":"3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342"}
May 29 10:27:41.715337 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.715337    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l" event={"ID":"05c7df0a-f480-4f7f-8801-53c8e08172e3","Type":"ContainerStarted","Data":"4acb8bdabe24ab4026bf5c6c9f21f394f033677057408eeec9dd85f556e3307a"}
May 29 10:27:41.730468 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:41.730422    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l" podStartSLOduration=1.730408263 podStartE2EDuration="1.730408263s" podCreationTimestamp="2026-05-29 10:27:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:27:41.728847382 +0000 UTC m=+884.625407000" watchObservedRunningTime="2026-05-29 10:27:41.730408263 +0000 UTC m=+884.626967883"
May 29 10:27:46.751519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.751487    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"]
May 29 10:27:46.755086 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.755067    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:46.758114 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.758092    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 29 10:27:46.758237 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.758090    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-g8cdj\""
May 29 10:27:46.758905 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.758891    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 29 10:27:46.758954 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.758918    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 29 10:27:46.772149 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.772128    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"]
May 29 10:27:46.915941 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.915909    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfclq\" (UniqueName: \"kubernetes.io/projected/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-kube-api-access-jfclq\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:46.915941 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.915945    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-online-tls\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:46.916148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.916027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-feast-data\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:46.916148 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:46.916103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-registry-tls\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.016909 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.016809    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-registry-tls\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.016909 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.016872    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jfclq\" (UniqueName: \"kubernetes.io/projected/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-kube-api-access-jfclq\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.016909 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.016897    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-online-tls\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.017185 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.016972    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-feast-data\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.017351 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.017330    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-feast-data\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.019333 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.019303    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-online-tls\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.019446 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.019395    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-registry-tls\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.027276 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.027253    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfclq\" (UniqueName: \"kubernetes.io/projected/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-kube-api-access-jfclq\") pod \"feast-credit-scoring-5886d85b99-67gfj\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") " pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.064863 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.064833    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:47.208843 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.208812    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"]
May 29 10:27:47.211786 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:27:47.211761    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4814e0_a0bb_4d16_aafa_7d45a763dcb2.slice/crio-8ab1f2338e65d19d639d43a8dde7808336deeed39577521115410436509220c0 WatchSource:0}: Error finding container 8ab1f2338e65d19d639d43a8dde7808336deeed39577521115410436509220c0: Status 404 returned error can't find the container with id 8ab1f2338e65d19d639d43a8dde7808336deeed39577521115410436509220c0
May 29 10:27:47.735602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.735564    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerStarted","Data":"83f53f50afa51c29959721fb9721e987c9f9af72a7619c598550e723eb9a41d7"}
May 29 10:27:47.735602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:47.735603    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerStarted","Data":"8ab1f2338e65d19d639d43a8dde7808336deeed39577521115410436509220c0"}
May 29 10:27:48.740081 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:48.740047    2562 generic.go:358] "Generic (PLEG): container finished" podID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerID="83f53f50afa51c29959721fb9721e987c9f9af72a7619c598550e723eb9a41d7" exitCode=0
May 29 10:27:48.740484 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:48.740138    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerDied","Data":"83f53f50afa51c29959721fb9721e987c9f9af72a7619c598550e723eb9a41d7"}
May 29 10:27:49.744790 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:49.744757    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerStarted","Data":"f7a37d5165da6928ab7d6ce389ad6abe149c0290e349d3255f6716fe55116bf2"}
May 29 10:27:56.768042 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:56.768003    2562 generic.go:358] "Generic (PLEG): container finished" podID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerID="f7a37d5165da6928ab7d6ce389ad6abe149c0290e349d3255f6716fe55116bf2" exitCode=0
May 29 10:27:56.768392 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:56.768072    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerDied","Data":"f7a37d5165da6928ab7d6ce389ad6abe149c0290e349d3255f6716fe55116bf2"}
May 29 10:27:57.644701 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:57.644670    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:27:57.646129 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:57.646105    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:27:57.776421 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:57.776390    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerStarted","Data":"0738a5aa332e4c886c143562cfdddaac0940a87227db3c83d6b71f5a27c2cebb"}
May 29 10:27:57.776421 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:57.776426    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerStarted","Data":"4388fffe31ee55329e053c1b4adb82a7fb77f82c2251d6dc6f43afc0a09d54d8"}
May 29 10:27:57.798409 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:57.798345    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podStartSLOduration=11.798331731 podStartE2EDuration="11.798331731s" podCreationTimestamp="2026-05-29 10:27:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:27:57.796748953 +0000 UTC m=+900.693308572" watchObservedRunningTime="2026-05-29 10:27:57.798331731 +0000 UTC m=+900.694891350"
May 29 10:27:59.065460 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:59.065420    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:59.067717 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:59.065474    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:27:59.067717 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:59.067171    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 29 10:27:59.067717 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:27:59.067177    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused"
May 29 10:28:02.065473 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:02.065423    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused"
May 29 10:28:02.065473 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:02.065455    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 29 10:28:05.066046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:05.065998    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused"
May 29 10:28:05.071441 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:05.071414    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:28:05.071952 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:05.071929    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:28:05.075770 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:05.075748    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:28:07.065809 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:07.065755    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:28:08.066293 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:08.066265    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:28:08.822101 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:08.822075    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:28:12.516988 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.516954    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"]
May 29 10:28:12.522034 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.522015    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.524050 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.524026    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 29 10:28:12.524169 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.524033    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 29 10:28:12.524169 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.524032    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-72sjw\""
May 29 10:28:12.524169 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.524141    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 29 10:28:12.530318 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.530287    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"]
May 29 10:28:12.538824 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.538801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-online-tls\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.538912 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.538848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjk29\" (UniqueName: \"kubernetes.io/projected/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-kube-api-access-bjk29\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.538952 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.538910    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-registry-tls\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.538952 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.538932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-feast-data\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.640276 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.640236    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bjk29\" (UniqueName: \"kubernetes.io/projected/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-kube-api-access-bjk29\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.640434 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.640285    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-registry-tls\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.640434 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.640309    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-feast-data\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.640434 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.640381    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-online-tls\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.640814 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.640782    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-feast-data\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.642900 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.642880    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-registry-tls\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.642991 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.642916    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-online-tls\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.648366 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.648342    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjk29\" (UniqueName: \"kubernetes.io/projected/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-kube-api-access-bjk29\") pod \"feast-driver-ranking-85bc8777b-7c2p4\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") " pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.831980 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.831894    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:12.964099 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:12.963981    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"]
May 29 10:28:12.967106 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:28:12.967080    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0d08d4_61c0_46d2_a8a3_6a8c7bf23fcc.slice/crio-7d0408c659aea736903282be58ed1f6ab4e498fb8cf29eb92ed10089622a1c3a WatchSource:0}: Error finding container 7d0408c659aea736903282be58ed1f6ab4e498fb8cf29eb92ed10089622a1c3a: Status 404 returned error can't find the container with id 7d0408c659aea736903282be58ed1f6ab4e498fb8cf29eb92ed10089622a1c3a
May 29 10:28:13.838039 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:13.838000    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerStarted","Data":"2b56011e23c647bfa58c4f225627e9d2f2e616087acc52db7112e51794dca102"}
May 29 10:28:13.838411 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:13.838046    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerStarted","Data":"7d0408c659aea736903282be58ed1f6ab4e498fb8cf29eb92ed10089622a1c3a"}
May 29 10:28:17.850854 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:17.850815    2562 generic.go:358] "Generic (PLEG): container finished" podID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerID="2b56011e23c647bfa58c4f225627e9d2f2e616087acc52db7112e51794dca102" exitCode=0
May 29 10:28:17.851294 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:17.850892    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerDied","Data":"2b56011e23c647bfa58c4f225627e9d2f2e616087acc52db7112e51794dca102"}
May 29 10:28:18.855329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:18.855295    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerStarted","Data":"83827ceca47c2db237e32fcde0e8ad8a932354c659f6fa24ca68bb723b5c6d06"}
May 29 10:28:24.875414 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:24.875387    2562 generic.go:358] "Generic (PLEG): container finished" podID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerID="83827ceca47c2db237e32fcde0e8ad8a932354c659f6fa24ca68bb723b5c6d06" exitCode=0
May 29 10:28:24.875738 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:24.875465    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerDied","Data":"83827ceca47c2db237e32fcde0e8ad8a932354c659f6fa24ca68bb723b5c6d06"}
May 29 10:28:25.882071 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:25.882031    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerStarted","Data":"2f85f1071b553e1a4247fc21ae6bb0aa7b2c90f5e159c0ae88373dc74447ca25"}
May 29 10:28:25.882513 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:25.882080    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerStarted","Data":"b1b5714d0fdf0df66189fe95870d97d63dfa687a792f562a0d68c9be55eaf076"}
May 29 10:28:25.904417 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:25.904352    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podStartSLOduration=13.904333563 podStartE2EDuration="13.904333563s" podCreationTimestamp="2026-05-29 10:28:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:28:25.90414267 +0000 UTC m=+928.800702291" watchObservedRunningTime="2026-05-29 10:28:25.904333563 +0000 UTC m=+928.800893183"
May 29 10:28:27.833056 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:27.833019    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:27.833508 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:27.833067    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:27.834599 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:27.834571    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused"
May 29 10:28:27.834729 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:27.834592    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 29 10:28:30.832452 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:30.832399    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused"
May 29 10:28:31.055045 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:31.055011    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:31.055593 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:31.055569    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:31.058117 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:31.058064    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:32.832617 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:32.832561    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:33.832823 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:33.832762    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused"
May 29 10:28:36.832896 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:36.832863    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:28:36.833536 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:28:36.833517    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:29:51.785618 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:51.785587    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"]
May 29 10:29:51.786136 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:51.786009    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="registry" containerID="cri-o://4388fffe31ee55329e053c1b4adb82a7fb77f82c2251d6dc6f43afc0a09d54d8" gracePeriod=30
May 29 10:29:51.786136 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:51.786080    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="online" containerID="cri-o://0738a5aa332e4c886c143562cfdddaac0940a87227db3c83d6b71f5a27c2cebb" gracePeriod=30
May 29 10:29:51.794971 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:51.794942    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"]
May 29 10:29:51.795307 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:51.795253    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="registry" containerID="cri-o://b1b5714d0fdf0df66189fe95870d97d63dfa687a792f562a0d68c9be55eaf076" gracePeriod=30
May 29 10:29:51.795307 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:51.795281    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="online" containerID="cri-o://2f85f1071b553e1a4247fc21ae6bb0aa7b2c90f5e159c0ae88373dc74447ca25" gracePeriod=30
May 29 10:29:53.156945 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:53.156912    2562 generic.go:358] "Generic (PLEG): container finished" podID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerID="0738a5aa332e4c886c143562cfdddaac0940a87227db3c83d6b71f5a27c2cebb" exitCode=0
May 29 10:29:53.157385 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:53.156973    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerDied","Data":"0738a5aa332e4c886c143562cfdddaac0940a87227db3c83d6b71f5a27c2cebb"}
May 29 10:29:53.158968 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:53.158944    2562 generic.go:358] "Generic (PLEG): container finished" podID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerID="2f85f1071b553e1a4247fc21ae6bb0aa7b2c90f5e159c0ae88373dc74447ca25" exitCode=0
May 29 10:29:53.158968 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:53.158974    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerDied","Data":"2f85f1071b553e1a4247fc21ae6bb0aa7b2c90f5e159c0ae88373dc74447ca25"}
May 29 10:29:53.971412 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:53.971372    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-ps94l"]
May 29 10:29:53.975836 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:53.975806    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-5kbnn"]
May 29 10:29:53.976265 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:53.976214    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-5kbnn" podUID="4d7d606c-c643-4eb0-9df5-a38f2da4bfe5" containerName="redis" containerID="cri-o://c7676f26eade41b4c1c482280c3ba7ddb835a4f5e90d43dcd2b48faa56ea8ed2" gracePeriod=30
May 29 10:29:54.163838 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.163803    2562 generic.go:358] "Generic (PLEG): container finished" podID="4d7d606c-c643-4eb0-9df5-a38f2da4bfe5" containerID="c7676f26eade41b4c1c482280c3ba7ddb835a4f5e90d43dcd2b48faa56ea8ed2" exitCode=0
May 29 10:29:54.164271 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.163884    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-5kbnn" event={"ID":"4d7d606c-c643-4eb0-9df5-a38f2da4bfe5","Type":"ContainerDied","Data":"c7676f26eade41b4c1c482280c3ba7ddb835a4f5e90d43dcd2b48faa56ea8ed2"}
May 29 10:29:54.164271 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.164046    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l" podUID="05c7df0a-f480-4f7f-8801-53c8e08172e3" containerName="postgres" containerID="cri-o://3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342" gracePeriod=30
May 29 10:29:54.275064 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.275000    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-5kbnn"
May 29 10:29:54.334265 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.334233    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6mrh\" (UniqueName: \"kubernetes.io/projected/4d7d606c-c643-4eb0-9df5-a38f2da4bfe5-kube-api-access-l6mrh\") pod \"4d7d606c-c643-4eb0-9df5-a38f2da4bfe5\" (UID: \"4d7d606c-c643-4eb0-9df5-a38f2da4bfe5\") "
May 29 10:29:54.337840 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.337465    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7d606c-c643-4eb0-9df5-a38f2da4bfe5-kube-api-access-l6mrh" (OuterVolumeSpecName: "kube-api-access-l6mrh") pod "4d7d606c-c643-4eb0-9df5-a38f2da4bfe5" (UID: "4d7d606c-c643-4eb0-9df5-a38f2da4bfe5"). InnerVolumeSpecName "kube-api-access-l6mrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:29:54.410742 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.410719    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:29:54.435214 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.435188    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/05c7df0a-f480-4f7f-8801-53c8e08172e3-postgresdata\") pod \"05c7df0a-f480-4f7f-8801-53c8e08172e3\" (UID: \"05c7df0a-f480-4f7f-8801-53c8e08172e3\") "
May 29 10:29:54.435359 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.435253    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq7jb\" (UniqueName: \"kubernetes.io/projected/05c7df0a-f480-4f7f-8801-53c8e08172e3-kube-api-access-bq7jb\") pod \"05c7df0a-f480-4f7f-8801-53c8e08172e3\" (UID: \"05c7df0a-f480-4f7f-8801-53c8e08172e3\") "
May 29 10:29:54.435536 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.435520    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6mrh\" (UniqueName: \"kubernetes.io/projected/4d7d606c-c643-4eb0-9df5-a38f2da4bfe5-kube-api-access-l6mrh\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:29:54.437538 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.437508    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c7df0a-f480-4f7f-8801-53c8e08172e3-kube-api-access-bq7jb" (OuterVolumeSpecName: "kube-api-access-bq7jb") pod "05c7df0a-f480-4f7f-8801-53c8e08172e3" (UID: "05c7df0a-f480-4f7f-8801-53c8e08172e3"). InnerVolumeSpecName "kube-api-access-bq7jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:29:54.469325 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.469292    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7df0a-f480-4f7f-8801-53c8e08172e3-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "05c7df0a-f480-4f7f-8801-53c8e08172e3" (UID: "05c7df0a-f480-4f7f-8801-53c8e08172e3"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:29:54.536295 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.536209    2562 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/05c7df0a-f480-4f7f-8801-53c8e08172e3-postgresdata\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:29:54.536295 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:54.536244    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq7jb\" (UniqueName: \"kubernetes.io/projected/05c7df0a-f480-4f7f-8801-53c8e08172e3-kube-api-access-bq7jb\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:29:55.072942 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.072898    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 29 10:29:55.167519 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.167484    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-5kbnn" event={"ID":"4d7d606c-c643-4eb0-9df5-a38f2da4bfe5","Type":"ContainerDied","Data":"bbb422bd598f27e413553eea5d267e1460f24926db011f3ef502772ea325a495"}
May 29 10:29:55.167994 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.167534    2562 scope.go:117] "RemoveContainer" containerID="c7676f26eade41b4c1c482280c3ba7ddb835a4f5e90d43dcd2b48faa56ea8ed2"
May 29 10:29:55.167994 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.167496    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-5kbnn"
May 29 10:29:55.168695 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.168673    2562 generic.go:358] "Generic (PLEG): container finished" podID="05c7df0a-f480-4f7f-8801-53c8e08172e3" containerID="3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342" exitCode=0
May 29 10:29:55.168760 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.168710    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l"
May 29 10:29:55.168760 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.168726    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l" event={"ID":"05c7df0a-f480-4f7f-8801-53c8e08172e3","Type":"ContainerDied","Data":"3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342"}
May 29 10:29:55.168760 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.168750    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-ps94l" event={"ID":"05c7df0a-f480-4f7f-8801-53c8e08172e3","Type":"ContainerDied","Data":"4acb8bdabe24ab4026bf5c6c9f21f394f033677057408eeec9dd85f556e3307a"}
May 29 10:29:55.175463 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.175448    2562 scope.go:117] "RemoveContainer" containerID="3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342"
May 29 10:29:55.182865 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.182837    2562 scope.go:117] "RemoveContainer" containerID="3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342"
May 29 10:29:55.183091 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:29:55.183075    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342\": container with ID starting with 3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342 not found: ID does not exist" containerID="3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342"
May 29 10:29:55.183137 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.183098    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342"} err="failed to get container status \"3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342\": rpc error: code = NotFound desc = could not find container \"3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342\": container with ID starting with 3fec9673348530300ba1f9bd2206644162f702f37500d4a188a3405f2d2df342 not found: ID does not exist"
May 29 10:29:55.188337 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.188316    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-5kbnn"]
May 29 10:29:55.194003 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.193982    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-5kbnn"]
May 29 10:29:55.201092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.201068    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-ps94l"]
May 29 10:29:55.203370 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.203351    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-ps94l"]
May 29 10:29:55.703369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.703337    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05c7df0a-f480-4f7f-8801-53c8e08172e3" path="/var/lib/kubelet/pods/05c7df0a-f480-4f7f-8801-53c8e08172e3/volumes"
May 29 10:29:55.703683 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:29:55.703669    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d7d606c-c643-4eb0-9df5-a38f2da4bfe5" path="/var/lib/kubelet/pods/4d7d606c-c643-4eb0-9df5-a38f2da4bfe5/volumes"
May 29 10:30:01.056502 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:01.056457    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 29 10:30:05.072688 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:05.072601    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 29 10:30:11.056730 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:11.056683    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 29 10:30:15.073020 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:15.072969    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 29 10:30:15.073423 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:15.073146    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:30:21.056217 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:21.056174    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 29 10:30:21.056628 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:21.056315    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:30:22.261457 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.261425    2562 generic.go:358] "Generic (PLEG): container finished" podID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerID="4388fffe31ee55329e053c1b4adb82a7fb77f82c2251d6dc6f43afc0a09d54d8" exitCode=137
May 29 10:30:22.261949 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.261508    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerDied","Data":"4388fffe31ee55329e053c1b4adb82a7fb77f82c2251d6dc6f43afc0a09d54d8"}
May 29 10:30:22.266498 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.266474    2562 generic.go:358] "Generic (PLEG): container finished" podID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerID="b1b5714d0fdf0df66189fe95870d97d63dfa687a792f562a0d68c9be55eaf076" exitCode=137
May 29 10:30:22.266617 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.266508    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerDied","Data":"b1b5714d0fdf0df66189fe95870d97d63dfa687a792f562a0d68c9be55eaf076"}
May 29 10:30:22.429198 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.429177    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:30:22.459038 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.459014    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:30:22.596394 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.596364    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-feast-data\") pod \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") "
May 29 10:30:22.596754 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.596409    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-feast-data\") pod \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") "
May 29 10:30:22.596754 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.596445    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-registry-tls\") pod \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") "
May 29 10:30:22.596754 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.596473    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-online-tls\") pod \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") "
May 29 10:30:22.596754 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.596551    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfclq\" (UniqueName: \"kubernetes.io/projected/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-kube-api-access-jfclq\") pod \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\" (UID: \"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2\") "
May 29 10:30:22.596754 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.596578    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjk29\" (UniqueName: \"kubernetes.io/projected/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-kube-api-access-bjk29\") pod \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") "
May 29 10:30:22.596754 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.596621    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-online-tls\") pod \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") "
May 29 10:30:22.597082 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.596939    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-registry-tls\") pod \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\" (UID: \"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc\") "
May 29 10:30:22.597240 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.596944    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-feast-data" (OuterVolumeSpecName: "feast-data") pod "9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" (UID: "9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:30:22.599250 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.599224    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" (UID: "9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:30:22.599367 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.599332    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-kube-api-access-bjk29" (OuterVolumeSpecName: "kube-api-access-bjk29") pod "9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" (UID: "9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc"). InnerVolumeSpecName "kube-api-access-bjk29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:30:22.599422 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.599378    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-kube-api-access-jfclq" (OuterVolumeSpecName: "kube-api-access-jfclq") pod "5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" (UID: "5a4814e0-a0bb-4d16-aafa-7d45a763dcb2"). InnerVolumeSpecName "kube-api-access-jfclq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:30:22.599466 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.599414    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-online-tls" (OuterVolumeSpecName: "online-tls") pod "5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" (UID: "5a4814e0-a0bb-4d16-aafa-7d45a763dcb2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:30:22.599501 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.599462    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-online-tls" (OuterVolumeSpecName: "online-tls") pod "9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" (UID: "9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:30:22.599583 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.599564    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" (UID: "5a4814e0-a0bb-4d16-aafa-7d45a763dcb2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:30:22.607396 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.607374    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-feast-data" (OuterVolumeSpecName: "feast-data") pod "5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" (UID: "5a4814e0-a0bb-4d16-aafa-7d45a763dcb2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:30:22.698087 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.698054    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfclq\" (UniqueName: \"kubernetes.io/projected/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-kube-api-access-jfclq\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:30:22.698087 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.698081    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjk29\" (UniqueName: \"kubernetes.io/projected/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-kube-api-access-bjk29\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:30:22.698087 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.698092    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:30:22.698300 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.698102    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:30:22.698300 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.698111    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:30:22.698300 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.698119    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:30:22.698300 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.698126    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:30:22.698300 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:22.698134    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:30:23.271407 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.271374    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj" event={"ID":"5a4814e0-a0bb-4d16-aafa-7d45a763dcb2","Type":"ContainerDied","Data":"8ab1f2338e65d19d639d43a8dde7808336deeed39577521115410436509220c0"}
May 29 10:30:23.271407 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.271407    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"
May 29 10:30:23.271919 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.271414    2562 scope.go:117] "RemoveContainer" containerID="0738a5aa332e4c886c143562cfdddaac0940a87227db3c83d6b71f5a27c2cebb"
May 29 10:30:23.273540 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.273524    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"
May 29 10:30:23.273654 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.273539    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4" event={"ID":"9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc","Type":"ContainerDied","Data":"7d0408c659aea736903282be58ed1f6ab4e498fb8cf29eb92ed10089622a1c3a"}
May 29 10:30:23.280883 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.280861    2562 scope.go:117] "RemoveContainer" containerID="4388fffe31ee55329e053c1b4adb82a7fb77f82c2251d6dc6f43afc0a09d54d8"
May 29 10:30:23.288587 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.288572    2562 scope.go:117] "RemoveContainer" containerID="f7a37d5165da6928ab7d6ce389ad6abe149c0290e349d3255f6716fe55116bf2"
May 29 10:30:23.295963 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.295923    2562 scope.go:117] "RemoveContainer" containerID="83f53f50afa51c29959721fb9721e987c9f9af72a7619c598550e723eb9a41d7"
May 29 10:30:23.297367 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.297315    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"]
May 29 10:30:23.301973 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.301947    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5886d85b99-67gfj"]
May 29 10:30:23.304043 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.304031    2562 scope.go:117] "RemoveContainer" containerID="2f85f1071b553e1a4247fc21ae6bb0aa7b2c90f5e159c0ae88373dc74447ca25"
May 29 10:30:23.311451 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.311430    2562 scope.go:117] "RemoveContainer" containerID="b1b5714d0fdf0df66189fe95870d97d63dfa687a792f562a0d68c9be55eaf076"
May 29 10:30:23.312002 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.311983    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"]
May 29 10:30:23.315543 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.315524    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-85bc8777b-7c2p4"]
May 29 10:30:23.319263 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.319247    2562 scope.go:117] "RemoveContainer" containerID="83827ceca47c2db237e32fcde0e8ad8a932354c659f6fa24ca68bb723b5c6d06"
May 29 10:30:23.326062 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.326043    2562 scope.go:117] "RemoveContainer" containerID="2b56011e23c647bfa58c4f225627e9d2f2e616087acc52db7112e51794dca102"
May 29 10:30:23.703897 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.703864    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" path="/var/lib/kubelet/pods/5a4814e0-a0bb-4d16-aafa-7d45a763dcb2/volumes"
May 29 10:30:23.704369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:23.704356    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" path="/var/lib/kubelet/pods/9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc/volumes"
May 29 10:30:25.023103 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023071    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"]
May 29 10:30:25.023471 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023430    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="feast-init"
May 29 10:30:25.023471 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023450    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="feast-init"
May 29 10:30:25.023471 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023469    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="feast-apply"
May 29 10:30:25.023570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023480    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="feast-apply"
May 29 10:30:25.023570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023500    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7d606c-c643-4eb0-9df5-a38f2da4bfe5" containerName="redis"
May 29 10:30:25.023570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023507    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7d606c-c643-4eb0-9df5-a38f2da4bfe5" containerName="redis"
May 29 10:30:25.023570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023517    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="online"
May 29 10:30:25.023570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023522    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="online"
May 29 10:30:25.023570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023528    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="registry"
May 29 10:30:25.023570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023536    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="registry"
May 29 10:30:25.023570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023554    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="online"
May 29 10:30:25.023570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023565    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="online"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023578    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="feast-apply"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023589    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="feast-apply"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023602    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c7df0a-f480-4f7f-8801-53c8e08172e3" containerName="postgres"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023612    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c7df0a-f480-4f7f-8801-53c8e08172e3" containerName="postgres"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023619    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="registry"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023629    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="registry"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023650    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="feast-init"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023660    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="feast-init"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023781    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="4d7d606c-c643-4eb0-9df5-a38f2da4bfe5" containerName="redis"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023789    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="online"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023799    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="registry"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023810    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="05c7df0a-f480-4f7f-8801-53c8e08172e3" containerName="postgres"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023821    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="9d0d08d4-61c0-46d2-a8a3-6a8c7bf23fcc" containerName="registry"
May 29 10:30:25.023859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.023839    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="5a4814e0-a0bb-4d16-aafa-7d45a763dcb2" containerName="online"
May 29 10:30:25.029434 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.029414    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.031166 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.031140    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 29 10:30:25.031375 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.031359    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 29 10:30:25.031622 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.031610    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 10:30:25.031702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.031618    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-6gln6\""
May 29 10:30:25.031772 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.031761    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 10:30:25.031932 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.031915    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 29 10:30:25.032135 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.032116    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 29 10:30:25.037986 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.037967    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"]
May 29 10:30:25.119929 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.119887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.119929 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.119930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/41d6d803-ddae-46d6-8af9-8a7fe49a197a-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.120127 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.119959    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.120127 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.120014    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrn9\" (UniqueName: \"kubernetes.io/projected/41d6d803-ddae-46d6-8af9-8a7fe49a197a-kube-api-access-fgrn9\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.120127 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.120068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.120127 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.120099    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.221448 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.221297    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.221448 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.221359    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.221448 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.221428    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.221783 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.221478    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/41d6d803-ddae-46d6-8af9-8a7fe49a197a-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.221783 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.221507    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.221783 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.221533    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrn9\" (UniqueName: \"kubernetes.io/projected/41d6d803-ddae-46d6-8af9-8a7fe49a197a-kube-api-access-fgrn9\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.221783 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:30:25.221761    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 29 10:30:25.221954 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:30:25.221848    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-offline-tls podName:41d6d803-ddae-46d6-8af9-8a7fe49a197a nodeName:}" failed. No retries permitted until 2026-05-29 10:30:25.721823832 +0000 UTC m=+1048.618383447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-7wk5j" (UID: "41d6d803-ddae-46d6-8af9-8a7fe49a197a") : secret "feast-simple-feast-setup-offline-tls" not found
May 29 10:30:25.222030 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.222006    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/41d6d803-ddae-46d6-8af9-8a7fe49a197a-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.224233 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.224204    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.224348 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.224214    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.224348 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.224312    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.230429 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.230406    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrn9\" (UniqueName: \"kubernetes.io/projected/41d6d803-ddae-46d6-8af9-8a7fe49a197a-kube-api-access-fgrn9\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.726787 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.726754    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.729237 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.729218    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-7wk5j\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:25.952525 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:25.952485    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:26.082536 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:26.082508    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"]
May 29 10:30:26.084559 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:30:26.084534    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41d6d803_ddae_46d6_8af9_8a7fe49a197a.slice/crio-103e8455ff6b1c897e0e7984b35544c8d5def101f3dedb91645abfd5648e09cb WatchSource:0}: Error finding container 103e8455ff6b1c897e0e7984b35544c8d5def101f3dedb91645abfd5648e09cb: Status 404 returned error can't find the container with id 103e8455ff6b1c897e0e7984b35544c8d5def101f3dedb91645abfd5648e09cb
May 29 10:30:26.086277 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:26.086262    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 10:30:26.284570 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:26.284480    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerStarted","Data":"103e8455ff6b1c897e0e7984b35544c8d5def101f3dedb91645abfd5648e09cb"}
May 29 10:30:41.342959 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:41.342929    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerStarted","Data":"4d55e5562b6bda03a64d9dd591d5f5e66ae5642638ef09379faccadb864f0ac3"}
May 29 10:30:46.359597 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:46.359564    2562 generic.go:358] "Generic (PLEG): container finished" podID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerID="4d55e5562b6bda03a64d9dd591d5f5e66ae5642638ef09379faccadb864f0ac3" exitCode=0
May 29 10:30:46.359975 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:46.359626    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerDied","Data":"4d55e5562b6bda03a64d9dd591d5f5e66ae5642638ef09379faccadb864f0ac3"}
May 29 10:30:47.364563 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:47.364527    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerStarted","Data":"57d71a4b3bfc411854f0fde90ee4385a4b5187b0884eeaacf18adeb405df34cc"}
May 29 10:30:52.381214 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:52.381178    2562 generic.go:358] "Generic (PLEG): container finished" podID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerID="57d71a4b3bfc411854f0fde90ee4385a4b5187b0884eeaacf18adeb405df34cc" exitCode=0
May 29 10:30:52.381591 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:52.381253    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerDied","Data":"57d71a4b3bfc411854f0fde90ee4385a4b5187b0884eeaacf18adeb405df34cc"}
May 29 10:30:53.389287 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:53.389244    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerStarted","Data":"eed6b7885881aa89f274fb87743e328d82b416bbaf2ebfdcbd814c2d5d0ee7e4"}
May 29 10:30:53.389906 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:53.389300    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerStarted","Data":"a9604d2f107b6fe0079132487b93920d6665d5edf4612bc6e6ac2dd36c9bc135"}
May 29 10:30:53.389906 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:53.389317    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerStarted","Data":"8040966bf7b75ed530a5f90efe6e25574cb2e37c0030debc4cdb3d05912884ab"}
May 29 10:30:53.389906 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:53.389330    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerStarted","Data":"5a584785238cab8107f900d7a5bb559c632ff13d400193bea05168e61906e4f6"}
May 29 10:30:53.412085 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:53.412010    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podStartSLOduration=13.226409224 podStartE2EDuration="28.411988087s" podCreationTimestamp="2026-05-29 10:30:25 +0000 UTC" firstStartedPulling="2026-05-29 10:30:26.086411095 +0000 UTC m=+1048.982970722" lastFinishedPulling="2026-05-29 10:30:41.271989985 +0000 UTC m=+1064.168549585" observedRunningTime="2026-05-29 10:30:53.411045795 +0000 UTC m=+1076.307605414" watchObservedRunningTime="2026-05-29 10:30:53.411988087 +0000 UTC m=+1076.308547706"
May 29 10:30:55.953951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.952943    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:55.953951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.952992    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:55.953951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.953006    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:55.953951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.953019    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:55.953951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.953030    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:55.953951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.953042    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:55.953951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.953054    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:55.954778 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.954625    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 29 10:30:55.954778 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.954683    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused"
May 29 10:30:55.954778 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.954627    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused"
May 29 10:30:55.955123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.955073    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:55.955123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:55.955107    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused"
May 29 10:30:58.954075 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:58.954044    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:58.954537 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:58.954420    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:58.954901 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:58.954879    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:58.954994 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:58.954935    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:58.955774 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:58.955754    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:58.958869 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:58.958848    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:59.411927 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:59.411897    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:30:59.414269 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:30:59.414250    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:32:02.296996 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.296964    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"]
May 29 10:32:02.297599 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.297327    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="registry" containerID="cri-o://5a584785238cab8107f900d7a5bb559c632ff13d400193bea05168e61906e4f6" gracePeriod=30
May 29 10:32:02.297599 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.297375    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="ui" containerID="cri-o://eed6b7885881aa89f274fb87743e328d82b416bbaf2ebfdcbd814c2d5d0ee7e4" gracePeriod=30
May 29 10:32:02.297599 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.297425    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="offline" containerID="cri-o://a9604d2f107b6fe0079132487b93920d6665d5edf4612bc6e6ac2dd36c9bc135" gracePeriod=30
May 29 10:32:02.297599 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.297460    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="online" containerID="cri-o://8040966bf7b75ed530a5f90efe6e25574cb2e37c0030debc4cdb3d05912884ab" gracePeriod=30
May 29 10:32:02.738578 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.738531    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"]
May 29 10:32:02.752304 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.752259    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"]
May 29 10:32:02.752547 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.752525    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.754426 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.754401    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-5b7tj\""
May 29 10:32:02.787271 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.787231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.787451 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.787304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.787451 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.787339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.787451 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.787415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwt24\" (UniqueName: \"kubernetes.io/projected/af791824-7fb4-4d56-b945-a2ac38b040c3-kube-api-access-rwt24\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.787451 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.787442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.787723 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.787465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af791824-7fb4-4d56-b945-a2ac38b040c3-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.888073 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.888032    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.888073 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.888082    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rwt24\" (UniqueName: \"kubernetes.io/projected/af791824-7fb4-4d56-b945-a2ac38b040c3-kube-api-access-rwt24\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.888353 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.888117    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.888353 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.888138    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af791824-7fb4-4d56-b945-a2ac38b040c3-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.888353 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.888160    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.888353 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.888209    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.888353 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:32:02.888209    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 29 10:32:02.888353 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:32:02.888315    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-offline-tls podName:af791824-7fb4-4d56-b945-a2ac38b040c3 nodeName:}" failed. No retries permitted until 2026-05-29 10:32:03.38829133 +0000 UTC m=+1146.284850941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-p4vpp" (UID: "af791824-7fb4-4d56-b945-a2ac38b040c3") : secret "feast-simple-feast-setup-offline-tls" not found
May 29 10:32:02.888353 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:32:02.888334    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 29 10:32:02.888713 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:32:02.888391    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-online-tls podName:af791824-7fb4-4d56-b945-a2ac38b040c3 nodeName:}" failed. No retries permitted until 2026-05-29 10:32:03.388379304 +0000 UTC m=+1146.284938904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-online-tls") pod "feast-simple-feast-setup-664fcfcc58-p4vpp" (UID: "af791824-7fb4-4d56-b945-a2ac38b040c3") : secret "feast-simple-feast-setup-online-tls" not found
May 29 10:32:02.888713 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.888573    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af791824-7fb4-4d56-b945-a2ac38b040c3-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.890805 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.890779    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.890949 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.890826    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:02.896134 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:02.896108    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwt24\" (UniqueName: \"kubernetes.io/projected/af791824-7fb4-4d56-b945-a2ac38b040c3-kube-api-access-rwt24\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:03.392287 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.392255    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:03.392705 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.392322    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:03.394876 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.394856    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:03.394930 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.394907    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p4vpp\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:03.604665 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.604615    2562 generic.go:358] "Generic (PLEG): container finished" podID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerID="eed6b7885881aa89f274fb87743e328d82b416bbaf2ebfdcbd814c2d5d0ee7e4" exitCode=0
May 29 10:32:03.604665 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.604656    2562 generic.go:358] "Generic (PLEG): container finished" podID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerID="8040966bf7b75ed530a5f90efe6e25574cb2e37c0030debc4cdb3d05912884ab" exitCode=0
May 29 10:32:03.604864 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.604691    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerDied","Data":"eed6b7885881aa89f274fb87743e328d82b416bbaf2ebfdcbd814c2d5d0ee7e4"}
May 29 10:32:03.604864 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.604719    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerDied","Data":"8040966bf7b75ed530a5f90efe6e25574cb2e37c0030debc4cdb3d05912884ab"}
May 29 10:32:03.662602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.662530    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:03.794879 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:03.794852    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"]
May 29 10:32:03.797854 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:32:03.797827    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf791824_7fb4_4d56_b945_a2ac38b040c3.slice/crio-18126ef6d9ac2cc92688a22741f736e54e1cd92b474b8d2b58133ab5d58adda9 WatchSource:0}: Error finding container 18126ef6d9ac2cc92688a22741f736e54e1cd92b474b8d2b58133ab5d58adda9: Status 404 returned error can't find the container with id 18126ef6d9ac2cc92688a22741f736e54e1cd92b474b8d2b58133ab5d58adda9
May 29 10:32:04.609400 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:04.609345    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerStarted","Data":"9efa8c8d8a96b3c064190d7b9ce00280737988dd1b503757b123b370587546fd"}
May 29 10:32:04.609774 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:04.609407    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerStarted","Data":"18126ef6d9ac2cc92688a22741f736e54e1cd92b474b8d2b58133ab5d58adda9"}
May 29 10:32:08.624680 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:08.624575    2562 generic.go:358] "Generic (PLEG): container finished" podID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerID="9efa8c8d8a96b3c064190d7b9ce00280737988dd1b503757b123b370587546fd" exitCode=0
May 29 10:32:08.624680 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:08.624620    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerDied","Data":"9efa8c8d8a96b3c064190d7b9ce00280737988dd1b503757b123b370587546fd"}
May 29 10:32:09.412214 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:09.412163    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused"
May 29 10:32:09.412214 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:09.412210    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 29 10:32:09.629322 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:09.629293    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerStarted","Data":"5fc89bd765f1831fcfcec12b9b986a47dc8f3fd940d82f8427539cb826ece64e"}
May 29 10:32:14.653656 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:14.653603    2562 generic.go:358] "Generic (PLEG): container finished" podID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerID="5fc89bd765f1831fcfcec12b9b986a47dc8f3fd940d82f8427539cb826ece64e" exitCode=0
May 29 10:32:14.654025 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:14.653668    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerDied","Data":"5fc89bd765f1831fcfcec12b9b986a47dc8f3fd940d82f8427539cb826ece64e"}
May 29 10:32:15.662183 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.662140    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerStarted","Data":"8fafbba2fb7bbf188ac8abdb2a3d017d7b21f654a4baffe5a687f2f15a95eef1"}
May 29 10:32:15.662676 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.662196    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerStarted","Data":"136a7241407bef25c096d1198dcbed02584d845b8ec4d3ea6fbec732a4ae6725"}
May 29 10:32:15.662676 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.662220    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerStarted","Data":"ab40d064d7d2313f3cf9b1acfccd8e1f98259cad388a6720378d5907a230e558"}
May 29 10:32:15.662676 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.662233    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerStarted","Data":"0c6d867f6de100748138677e667fef8acf70c356f87474afba3a23ea5adfe3b7"}
May 29 10:32:15.663080 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.663054    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:15.663172 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.663086    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:15.663172 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.663100    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:15.663172 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.663113    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:15.664453 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.664418    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 29 10:32:15.664552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.664435    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused"
May 29 10:32:15.664552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.664428    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused"
May 29 10:32:15.664697 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.664546    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused"
May 29 10:32:15.689588 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:15.689190    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podStartSLOduration=13.689172218 podStartE2EDuration="13.689172218s" podCreationTimestamp="2026-05-29 10:32:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:32:15.68629984 +0000 UTC m=+1158.582859483" watchObservedRunningTime="2026-05-29 10:32:15.689172218 +0000 UTC m=+1158.585731856"
May 29 10:32:18.663829 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:18.663778    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused"
May 29 10:32:18.664296 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:18.663778    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused"
May 29 10:32:18.664464 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:18.664427    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused"
May 29 10:32:18.664464 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:18.664450    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 29 10:32:19.412270 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:19.412222    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused"
May 29 10:32:19.412270 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:19.412257    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 29 10:32:21.663992 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.663961    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.664458 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.664085    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.664458 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.664296    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.664676 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.664575    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.664676 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.664603    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.664676 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.664618    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.664816 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.664776    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.665071 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.665048    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.665151 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.665143    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.668594 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.668575    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.683916 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.683846    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:21.687503 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:21.687485    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:32:29.412143 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:29.412099    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 29 10:32:29.412547 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:29.412100    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused"
May 29 10:32:29.412547 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:29.412220    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:32:29.412547 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:29.412332    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:32:32.727123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:32.727092    2562 generic.go:358] "Generic (PLEG): container finished" podID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerID="a9604d2f107b6fe0079132487b93920d6665d5edf4612bc6e6ac2dd36c9bc135" exitCode=137
May 29 10:32:32.727123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:32.727118    2562 generic.go:358] "Generic (PLEG): container finished" podID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerID="5a584785238cab8107f900d7a5bb559c632ff13d400193bea05168e61906e4f6" exitCode=137
May 29 10:32:32.727537 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:32.727163    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerDied","Data":"a9604d2f107b6fe0079132487b93920d6665d5edf4612bc6e6ac2dd36c9bc135"}
May 29 10:32:32.727537 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:32.727200    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerDied","Data":"5a584785238cab8107f900d7a5bb559c632ff13d400193bea05168e61906e4f6"}
May 29 10:32:32.945725 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:32.945699    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:32:33.075288 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.075188    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-offline-tls\") pod \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") "
May 29 10:32:33.075288 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.075247    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-registry-tls\") pod \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") "
May 29 10:32:33.075518 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.075300    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-ui-tls\") pod \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") "
May 29 10:32:33.075518 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.075325    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/41d6d803-ddae-46d6-8af9-8a7fe49a197a-feast-data\") pod \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") "
May 29 10:32:33.075518 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.075350    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgrn9\" (UniqueName: \"kubernetes.io/projected/41d6d803-ddae-46d6-8af9-8a7fe49a197a-kube-api-access-fgrn9\") pod \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") "
May 29 10:32:33.075518 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.075382    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-online-tls\") pod \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\" (UID: \"41d6d803-ddae-46d6-8af9-8a7fe49a197a\") "
May 29 10:32:33.075905 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.075878    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d6d803-ddae-46d6-8af9-8a7fe49a197a-feast-data" (OuterVolumeSpecName: "feast-data") pod "41d6d803-ddae-46d6-8af9-8a7fe49a197a" (UID: "41d6d803-ddae-46d6-8af9-8a7fe49a197a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:32:33.077941 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.077899    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "41d6d803-ddae-46d6-8af9-8a7fe49a197a" (UID: "41d6d803-ddae-46d6-8af9-8a7fe49a197a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:32:33.077941 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.077914    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-online-tls" (OuterVolumeSpecName: "online-tls") pod "41d6d803-ddae-46d6-8af9-8a7fe49a197a" (UID: "41d6d803-ddae-46d6-8af9-8a7fe49a197a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:32:33.078180 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.078143    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d6d803-ddae-46d6-8af9-8a7fe49a197a-kube-api-access-fgrn9" (OuterVolumeSpecName: "kube-api-access-fgrn9") pod "41d6d803-ddae-46d6-8af9-8a7fe49a197a" (UID: "41d6d803-ddae-46d6-8af9-8a7fe49a197a"). InnerVolumeSpecName "kube-api-access-fgrn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:32:33.078325 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.078282    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "41d6d803-ddae-46d6-8af9-8a7fe49a197a" (UID: "41d6d803-ddae-46d6-8af9-8a7fe49a197a"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:32:33.078588 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.078562    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "41d6d803-ddae-46d6-8af9-8a7fe49a197a" (UID: "41d6d803-ddae-46d6-8af9-8a7fe49a197a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:32:33.176707 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.176656    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgrn9\" (UniqueName: \"kubernetes.io/projected/41d6d803-ddae-46d6-8af9-8a7fe49a197a-kube-api-access-fgrn9\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:32:33.176707 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.176700    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:32:33.176707 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.176710    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:32:33.176707 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.176718    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:32:33.176707 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.176726    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/41d6d803-ddae-46d6-8af9-8a7fe49a197a-ui-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:32:33.177000 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.176735    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/41d6d803-ddae-46d6-8af9-8a7fe49a197a-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:32:33.732731 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.732695    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j" event={"ID":"41d6d803-ddae-46d6-8af9-8a7fe49a197a","Type":"ContainerDied","Data":"103e8455ff6b1c897e0e7984b35544c8d5def101f3dedb91645abfd5648e09cb"}
May 29 10:32:33.733094 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.732747    2562 scope.go:117] "RemoveContainer" containerID="eed6b7885881aa89f274fb87743e328d82b416bbaf2ebfdcbd814c2d5d0ee7e4"
May 29 10:32:33.733094 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.732768    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"
May 29 10:32:33.741323 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.741298    2562 scope.go:117] "RemoveContainer" containerID="a9604d2f107b6fe0079132487b93920d6665d5edf4612bc6e6ac2dd36c9bc135"
May 29 10:32:33.749231 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.749210    2562 scope.go:117] "RemoveContainer" containerID="8040966bf7b75ed530a5f90efe6e25574cb2e37c0030debc4cdb3d05912884ab"
May 29 10:32:33.749753 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.749730    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"]
May 29 10:32:33.752258 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.752236    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-7wk5j"]
May 29 10:32:33.757395 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.757375    2562 scope.go:117] "RemoveContainer" containerID="5a584785238cab8107f900d7a5bb559c632ff13d400193bea05168e61906e4f6"
May 29 10:32:33.773658 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.773626    2562 scope.go:117] "RemoveContainer" containerID="57d71a4b3bfc411854f0fde90ee4385a4b5187b0884eeaacf18adeb405df34cc"
May 29 10:32:33.780509 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:33.780494    2562 scope.go:117] "RemoveContainer" containerID="4d55e5562b6bda03a64d9dd591d5f5e66ae5642638ef09379faccadb864f0ac3"
May 29 10:32:35.704162 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:35.704132    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" path="/var/lib/kubelet/pods/41d6d803-ddae-46d6-8af9-8a7fe49a197a/volumes"
May 29 10:32:57.668903 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:57.668870    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:32:57.677505 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:32:57.677485    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:33:24.431220 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431186    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"]
May 29 10:33:24.431701 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431684    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="offline"
May 29 10:33:24.431761 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431704    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="offline"
May 29 10:33:24.431761 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431717    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="ui"
May 29 10:33:24.431761 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431726    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="ui"
May 29 10:33:24.431761 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431739    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="feast-init"
May 29 10:33:24.431761 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431748    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="feast-init"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431764    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="online"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431772    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="online"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431793    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="registry"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431801    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="registry"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431816    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="feast-apply"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431823    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="feast-apply"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431894    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="ui"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431907    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="offline"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431919    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="registry"
May 29 10:33:24.432092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.431929    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="41d6d803-ddae-46d6-8af9-8a7fe49a197a" containerName="online"
May 29 10:33:24.435696 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.435676    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.437555 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.437534    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 29 10:33:24.437683 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.437586    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 29 10:33:24.437683 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.437604    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 29 10:33:24.437903 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.437888    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-dlb24\""
May 29 10:33:24.437958 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.437937    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 29 10:33:24.438011 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.437969    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 29 10:33:24.438011 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.437975    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 29 10:33:24.442998 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.442717    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"]
May 29 10:33:24.622352 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.622314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8b28b5a0-4228-4e8c-965f-984a3b683d87-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.622352 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.622352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wncj\" (UniqueName: \"kubernetes.io/projected/8b28b5a0-4228-4e8c-965f-984a3b683d87-kube-api-access-4wncj\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.622564 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.622375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.622564 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.622405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.622564 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.622442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.622564 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.622463    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28b5a0-4228-4e8c-965f-984a3b683d87-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.723252 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.723170    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28b5a0-4228-4e8c-965f-984a3b683d87-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.723252 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.723250    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8b28b5a0-4228-4e8c-965f-984a3b683d87-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.723436 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.723272    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4wncj\" (UniqueName: \"kubernetes.io/projected/8b28b5a0-4228-4e8c-965f-984a3b683d87-kube-api-access-4wncj\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.723436 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.723294    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.723436 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.723319    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.723436 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.723352    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.723722 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:33:24.723452    2562 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 29 10:33:24.723722 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:33:24.723455    2562 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 10:33:24.723722 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:33:24.723550    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-online-tls podName:8b28b5a0-4228-4e8c-965f-984a3b683d87 nodeName:}" failed. No retries permitted until 2026-05-29 10:33:25.223526931 +0000 UTC m=+1228.120086529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-online-tls") pod "feast-simple-feast-remote-setup-5cff94746b-zr7ns" (UID: "8b28b5a0-4228-4e8c-965f-984a3b683d87") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 10:33:24.723722 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:33:24.723578    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-ui-tls podName:8b28b5a0-4228-4e8c-965f-984a3b683d87 nodeName:}" failed. No retries permitted until 2026-05-29 10:33:25.223560184 +0000 UTC m=+1228.120119782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-ui-tls") pod "feast-simple-feast-remote-setup-5cff94746b-zr7ns" (UID: "8b28b5a0-4228-4e8c-965f-984a3b683d87") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 29 10:33:24.723722 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.723603    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28b5a0-4228-4e8c-965f-984a3b683d87-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.723946 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.723901    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8b28b5a0-4228-4e8c-965f-984a3b683d87-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.725859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.725841    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:24.731142 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:24.731122    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wncj\" (UniqueName: \"kubernetes.io/projected/8b28b5a0-4228-4e8c-965f-984a3b683d87-kube-api-access-4wncj\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:25.227617 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:25.227583    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:25.227843 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:25.227629    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:25.230083 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:25.230061    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:25.230212 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:25.230180    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-zr7ns\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:25.345390 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:25.345356    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:25.475481 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:25.475285    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"]
May 29 10:33:25.480087 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:33:25.479964    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b28b5a0_4228_4e8c_965f_984a3b683d87.slice/crio-2e8445f2f52a786f3d5510cef1cc0812bee351baf843a5f071fadf5ba9358c56 WatchSource:0}: Error finding container 2e8445f2f52a786f3d5510cef1cc0812bee351baf843a5f071fadf5ba9358c56: Status 404 returned error can't find the container with id 2e8445f2f52a786f3d5510cef1cc0812bee351baf843a5f071fadf5ba9358c56
May 29 10:33:25.887343 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:25.887304    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerStarted","Data":"7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb"}
May 29 10:33:25.887542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:25.887350    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerStarted","Data":"2e8445f2f52a786f3d5510cef1cc0812bee351baf843a5f071fadf5ba9358c56"}
May 29 10:33:30.902803 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:30.902766    2562 generic.go:358] "Generic (PLEG): container finished" podID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerID="7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb" exitCode=0
May 29 10:33:30.903213 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:30.902815    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerDied","Data":"7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb"}
May 29 10:33:31.908033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:31.907995    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerStarted","Data":"6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70"}
May 29 10:33:36.924444 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:36.924413    2562 generic.go:358] "Generic (PLEG): container finished" podID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerID="6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70" exitCode=0
May 29 10:33:36.924862 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:36.924496    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerDied","Data":"6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70"}
May 29 10:33:37.931265 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:37.931228    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerStarted","Data":"b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a"}
May 29 10:33:37.931751 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:37.931277    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerStarted","Data":"54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61"}
May 29 10:33:37.931751 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:37.931291    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerStarted","Data":"bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a"}
May 29 10:33:37.953380 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:37.953320    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podStartSLOduration=13.953302592 podStartE2EDuration="13.953302592s" podCreationTimestamp="2026-05-29 10:33:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:33:37.951122978 +0000 UTC m=+1240.847682622" watchObservedRunningTime="2026-05-29 10:33:37.953302592 +0000 UTC m=+1240.849862211"
May 29 10:33:40.346091 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:40.346054    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:40.346562 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:40.346147    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:40.346562 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:40.346164    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:40.347918 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:40.347869    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused"
May 29 10:33:40.347918 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:40.347893    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 29 10:33:40.348055 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:40.347895    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused"
May 29 10:33:43.347605 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:43.347577    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:43.348023 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:43.347715    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:43.348092 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:43.348076    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:43.348151 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:43.348101    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:43.348520 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:43.348499    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:43.348599 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:43.348557    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:43.351936 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:43.351916    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:43.951517 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:43.951482    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:33:43.954542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:33:43.954523    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:34:45.695023 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:45.694988    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"]
May 29 10:34:45.695563 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:45.695335    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="online" containerID="cri-o://bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a" gracePeriod=30
May 29 10:34:45.695563 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:45.695384    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="offline" containerID="cri-o://54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61" gracePeriod=30
May 29 10:34:45.695563 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:45.695399    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="ui" containerID="cri-o://b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a" gracePeriod=30
May 29 10:34:45.845702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:45.845668    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"]
May 29 10:34:45.846034 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:45.846006    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="registry" containerID="cri-o://0c6d867f6de100748138677e667fef8acf70c356f87474afba3a23ea5adfe3b7" gracePeriod=30
May 29 10:34:45.846122 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:45.846033    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="ui" containerID="cri-o://8fafbba2fb7bbf188ac8abdb2a3d017d7b21f654a4baffe5a687f2f15a95eef1" gracePeriod=30
May 29 10:34:45.846122 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:45.846049    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="online" containerID="cri-o://ab40d064d7d2313f3cf9b1acfccd8e1f98259cad388a6720378d5907a230e558" gracePeriod=30
May 29 10:34:45.846122 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:45.846076    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="offline" containerID="cri-o://136a7241407bef25c096d1198dcbed02584d845b8ec4d3ea6fbec732a4ae6725" gracePeriod=30
May 29 10:34:47.148557 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:47.148523    2562 generic.go:358] "Generic (PLEG): container finished" podID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerID="8fafbba2fb7bbf188ac8abdb2a3d017d7b21f654a4baffe5a687f2f15a95eef1" exitCode=0
May 29 10:34:47.148557 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:47.148550    2562 generic.go:358] "Generic (PLEG): container finished" podID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerID="ab40d064d7d2313f3cf9b1acfccd8e1f98259cad388a6720378d5907a230e558" exitCode=0
May 29 10:34:47.149036 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:47.148596    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerDied","Data":"8fafbba2fb7bbf188ac8abdb2a3d017d7b21f654a4baffe5a687f2f15a95eef1"}
May 29 10:34:47.149036 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:47.148654    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerDied","Data":"ab40d064d7d2313f3cf9b1acfccd8e1f98259cad388a6720378d5907a230e558"}
May 29 10:34:47.150990 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:47.150962    2562 generic.go:358] "Generic (PLEG): container finished" podID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerID="bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a" exitCode=0
May 29 10:34:47.151111 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:47.151007    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerDied","Data":"bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a"}
May 29 10:34:50.161970 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:50.161939    2562 generic.go:358] "Generic (PLEG): container finished" podID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerID="b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a" exitCode=0
May 29 10:34:50.162351 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:50.162014    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerDied","Data":"b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a"}
May 29 10:34:51.664948 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:51.664896    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused"
May 29 10:34:51.684531 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:51.684493    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 29 10:34:53.348318 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:53.348274    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused"
May 29 10:34:53.952602 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:34:53.952556    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 29 10:35:01.664982 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:01.664930    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused"
May 29 10:35:01.684676 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:01.684602    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 29 10:35:03.348354 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:03.348301    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused"
May 29 10:35:03.952606 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:03.952565    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 29 10:35:11.665317 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:11.665214    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused"
May 29 10:35:11.665742 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:11.665377    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:35:11.683947 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:11.683911    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 29 10:35:11.684112 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:11.684011    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:35:13.348585 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:13.348530    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused"
May 29 10:35:13.348989 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:13.348711    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:35:13.952531 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:13.952490    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 29 10:35:13.952734 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:13.952623    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:35:15.955904 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:15.955838    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:35:16.003326 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.003297    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-offline-tls\") pod \"8b28b5a0-4228-4e8c-965f-984a3b683d87\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") "
May 29 10:35:16.003491 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.003363    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wncj\" (UniqueName: \"kubernetes.io/projected/8b28b5a0-4228-4e8c-965f-984a3b683d87-kube-api-access-4wncj\") pod \"8b28b5a0-4228-4e8c-965f-984a3b683d87\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") "
May 29 10:35:16.003491 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.003394    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8b28b5a0-4228-4e8c-965f-984a3b683d87-registry-tls\") pod \"8b28b5a0-4228-4e8c-965f-984a3b683d87\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") "
May 29 10:35:16.003568 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.003545    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28b5a0-4228-4e8c-965f-984a3b683d87-feast-data\") pod \"8b28b5a0-4228-4e8c-965f-984a3b683d87\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") "
May 29 10:35:16.003629 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.003612    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-ui-tls\") pod \"8b28b5a0-4228-4e8c-965f-984a3b683d87\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") "
May 29 10:35:16.003707 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.003671    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-online-tls\") pod \"8b28b5a0-4228-4e8c-965f-984a3b683d87\" (UID: \"8b28b5a0-4228-4e8c-965f-984a3b683d87\") "
May 29 10:35:16.003766 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.003727    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b28b5a0-4228-4e8c-965f-984a3b683d87-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8b28b5a0-4228-4e8c-965f-984a3b683d87" (UID: "8b28b5a0-4228-4e8c-965f-984a3b683d87"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:35:16.003980 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.003957    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8b28b5a0-4228-4e8c-965f-984a3b683d87-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.004077 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.004002    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28b5a0-4228-4e8c-965f-984a3b683d87-feast-data" (OuterVolumeSpecName: "feast-data") pod "8b28b5a0-4228-4e8c-965f-984a3b683d87" (UID: "8b28b5a0-4228-4e8c-965f-984a3b683d87"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:35:16.005756 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.005728    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b28b5a0-4228-4e8c-965f-984a3b683d87-kube-api-access-4wncj" (OuterVolumeSpecName: "kube-api-access-4wncj") pod "8b28b5a0-4228-4e8c-965f-984a3b683d87" (UID: "8b28b5a0-4228-4e8c-965f-984a3b683d87"). InnerVolumeSpecName "kube-api-access-4wncj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:35:16.005859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.005763    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "8b28b5a0-4228-4e8c-965f-984a3b683d87" (UID: "8b28b5a0-4228-4e8c-965f-984a3b683d87"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:35:16.005859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.005788    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "8b28b5a0-4228-4e8c-965f-984a3b683d87" (UID: "8b28b5a0-4228-4e8c-965f-984a3b683d87"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:35:16.005859 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.005801    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-online-tls" (OuterVolumeSpecName: "online-tls") pod "8b28b5a0-4228-4e8c-965f-984a3b683d87" (UID: "8b28b5a0-4228-4e8c-965f-984a3b683d87"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:35:16.104969 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.104937    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28b5a0-4228-4e8c-965f-984a3b683d87-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.104969 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.104965    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-ui-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.104969 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.104973    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.104969 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.104981    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8b28b5a0-4228-4e8c-965f-984a3b683d87-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.105236 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.104991    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wncj\" (UniqueName: \"kubernetes.io/projected/8b28b5a0-4228-4e8c-965f-984a3b683d87-kube-api-access-4wncj\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.240943 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.240852    2562 generic.go:358] "Generic (PLEG): container finished" podID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerID="136a7241407bef25c096d1198dcbed02584d845b8ec4d3ea6fbec732a4ae6725" exitCode=137
May 29 10:35:16.240943 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.240876    2562 generic.go:358] "Generic (PLEG): container finished" podID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerID="0c6d867f6de100748138677e667fef8acf70c356f87474afba3a23ea5adfe3b7" exitCode=137
May 29 10:35:16.240943 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.240925    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerDied","Data":"136a7241407bef25c096d1198dcbed02584d845b8ec4d3ea6fbec732a4ae6725"}
May 29 10:35:16.241186 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.240966    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerDied","Data":"0c6d867f6de100748138677e667fef8acf70c356f87474afba3a23ea5adfe3b7"}
May 29 10:35:16.242844 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.242822    2562 generic.go:358] "Generic (PLEG): container finished" podID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerID="54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61" exitCode=137
May 29 10:35:16.242946 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.242890    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"
May 29 10:35:16.242985 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.242889    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerDied","Data":"54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61"}
May 29 10:35:16.243019 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.242984    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns" event={"ID":"8b28b5a0-4228-4e8c-965f-984a3b683d87","Type":"ContainerDied","Data":"2e8445f2f52a786f3d5510cef1cc0812bee351baf843a5f071fadf5ba9358c56"}
May 29 10:35:16.243019 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.243000    2562 scope.go:117] "RemoveContainer" containerID="b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a"
May 29 10:35:16.251921 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.251901    2562 scope.go:117] "RemoveContainer" containerID="54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61"
May 29 10:35:16.259822 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.259799    2562 scope.go:117] "RemoveContainer" containerID="bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a"
May 29 10:35:16.266112 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.266086    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"]
May 29 10:35:16.268270 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.268247    2562 scope.go:117] "RemoveContainer" containerID="6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70"
May 29 10:35:16.270873 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.270853    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-zr7ns"]
May 29 10:35:16.276909 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.276884    2562 scope.go:117] "RemoveContainer" containerID="7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb"
May 29 10:35:16.284390 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.284371    2562 scope.go:117] "RemoveContainer" containerID="b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a"
May 29 10:35:16.284684 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:35:16.284659    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a\": container with ID starting with b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a not found: ID does not exist" containerID="b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a"
May 29 10:35:16.284740 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.284697    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a"} err="failed to get container status \"b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a\": rpc error: code = NotFound desc = could not find container \"b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a\": container with ID starting with b61938f22ddac62ad31efcdc7425a01884cb07de5f8b0a50e604cc9598de657a not found: ID does not exist"
May 29 10:35:16.284740 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.284724    2562 scope.go:117] "RemoveContainer" containerID="54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61"
May 29 10:35:16.284937 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:35:16.284921    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61\": container with ID starting with 54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61 not found: ID does not exist" containerID="54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61"
May 29 10:35:16.284981 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.284942    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61"} err="failed to get container status \"54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61\": rpc error: code = NotFound desc = could not find container \"54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61\": container with ID starting with 54f1077df64b8023640d446a4ef4eccd22a9f00f11016520fa2b389662693c61 not found: ID does not exist"
May 29 10:35:16.284981 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.284958    2562 scope.go:117] "RemoveContainer" containerID="bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a"
May 29 10:35:16.285193 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:35:16.285175    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a\": container with ID starting with bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a not found: ID does not exist" containerID="bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a"
May 29 10:35:16.285231 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.285198    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a"} err="failed to get container status \"bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a\": rpc error: code = NotFound desc = could not find container \"bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a\": container with ID starting with bbfdb04e44b77b345990f145b6741c83f9c33ef84a3ddf7fa3386757227af24a not found: ID does not exist"
May 29 10:35:16.285231 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.285213    2562 scope.go:117] "RemoveContainer" containerID="6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70"
May 29 10:35:16.285412 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:35:16.285397    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70\": container with ID starting with 6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70 not found: ID does not exist" containerID="6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70"
May 29 10:35:16.285458 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.285416    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70"} err="failed to get container status \"6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70\": rpc error: code = NotFound desc = could not find container \"6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70\": container with ID starting with 6c96a1ccabe5ad18af80435f6a4405e2258741dbed22b19944dc10d39de0fa70 not found: ID does not exist"
May 29 10:35:16.285458 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.285436    2562 scope.go:117] "RemoveContainer" containerID="7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb"
May 29 10:35:16.285617 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:35:16.285601    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb\": container with ID starting with 7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb not found: ID does not exist" containerID="7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb"
May 29 10:35:16.285690 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.285620    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb"} err="failed to get container status \"7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb\": rpc error: code = NotFound desc = could not find container \"7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb\": container with ID starting with 7db7918bff1fd17727a2af2187ea739dee43bb2e503911dafaae006bb14e99fb not found: ID does not exist"
May 29 10:35:16.482690 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.482667    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:35:16.609257 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.609225    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af791824-7fb4-4d56-b945-a2ac38b040c3-feast-data\") pod \"af791824-7fb4-4d56-b945-a2ac38b040c3\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") "
May 29 10:35:16.609417 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.609279    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-registry-tls\") pod \"af791824-7fb4-4d56-b945-a2ac38b040c3\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") "
May 29 10:35:16.609417 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.609326    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-ui-tls\") pod \"af791824-7fb4-4d56-b945-a2ac38b040c3\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") "
May 29 10:35:16.609417 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.609344    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwt24\" (UniqueName: \"kubernetes.io/projected/af791824-7fb4-4d56-b945-a2ac38b040c3-kube-api-access-rwt24\") pod \"af791824-7fb4-4d56-b945-a2ac38b040c3\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") "
May 29 10:35:16.609591 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.609567    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-offline-tls\") pod \"af791824-7fb4-4d56-b945-a2ac38b040c3\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") "
May 29 10:35:16.609669 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.609618    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-online-tls\") pod \"af791824-7fb4-4d56-b945-a2ac38b040c3\" (UID: \"af791824-7fb4-4d56-b945-a2ac38b040c3\") "
May 29 10:35:16.609804 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.609780    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af791824-7fb4-4d56-b945-a2ac38b040c3-feast-data" (OuterVolumeSpecName: "feast-data") pod "af791824-7fb4-4d56-b945-a2ac38b040c3" (UID: "af791824-7fb4-4d56-b945-a2ac38b040c3"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:35:16.609941 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.609924    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af791824-7fb4-4d56-b945-a2ac38b040c3-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.611668 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.611613    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "af791824-7fb4-4d56-b945-a2ac38b040c3" (UID: "af791824-7fb4-4d56-b945-a2ac38b040c3"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:35:16.611668 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.611630    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "af791824-7fb4-4d56-b945-a2ac38b040c3" (UID: "af791824-7fb4-4d56-b945-a2ac38b040c3"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:35:16.611808 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.611671    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-online-tls" (OuterVolumeSpecName: "online-tls") pod "af791824-7fb4-4d56-b945-a2ac38b040c3" (UID: "af791824-7fb4-4d56-b945-a2ac38b040c3"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:35:16.612027 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.612009    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "af791824-7fb4-4d56-b945-a2ac38b040c3" (UID: "af791824-7fb4-4d56-b945-a2ac38b040c3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:35:16.612072 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.612020    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af791824-7fb4-4d56-b945-a2ac38b040c3-kube-api-access-rwt24" (OuterVolumeSpecName: "kube-api-access-rwt24") pod "af791824-7fb4-4d56-b945-a2ac38b040c3" (UID: "af791824-7fb4-4d56-b945-a2ac38b040c3"). InnerVolumeSpecName "kube-api-access-rwt24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:35:16.711210 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.711177    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.711210 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.711204    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-ui-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.711210 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.711214    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwt24\" (UniqueName: \"kubernetes.io/projected/af791824-7fb4-4d56-b945-a2ac38b040c3-kube-api-access-rwt24\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.711420 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.711222    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:16.711420 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:16.711231    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af791824-7fb4-4d56-b945-a2ac38b040c3-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:35:17.248393 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.248353    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp" event={"ID":"af791824-7fb4-4d56-b945-a2ac38b040c3","Type":"ContainerDied","Data":"18126ef6d9ac2cc92688a22741f736e54e1cd92b474b8d2b58133ab5d58adda9"}
May 29 10:35:17.248882 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.248406    2562 scope.go:117] "RemoveContainer" containerID="8fafbba2fb7bbf188ac8abdb2a3d017d7b21f654a4baffe5a687f2f15a95eef1"
May 29 10:35:17.248882 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.248416    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"
May 29 10:35:17.258542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.258522    2562 scope.go:117] "RemoveContainer" containerID="136a7241407bef25c096d1198dcbed02584d845b8ec4d3ea6fbec732a4ae6725"
May 29 10:35:17.266123 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.266102    2562 scope.go:117] "RemoveContainer" containerID="ab40d064d7d2313f3cf9b1acfccd8e1f98259cad388a6720378d5907a230e558"
May 29 10:35:17.270424 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.270395    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"]
May 29 10:35:17.273234 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.273211    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p4vpp"]
May 29 10:35:17.274756 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.274660    2562 scope.go:117] "RemoveContainer" containerID="0c6d867f6de100748138677e667fef8acf70c356f87474afba3a23ea5adfe3b7"
May 29 10:35:17.282235 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.282214    2562 scope.go:117] "RemoveContainer" containerID="5fc89bd765f1831fcfcec12b9b986a47dc8f3fd940d82f8427539cb826ece64e"
May 29 10:35:17.289547 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.289529    2562 scope.go:117] "RemoveContainer" containerID="9efa8c8d8a96b3c064190d7b9ce00280737988dd1b503757b123b370587546fd"
May 29 10:35:17.705267 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.705232    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" path="/var/lib/kubelet/pods/8b28b5a0-4228-4e8c-965f-984a3b683d87/volumes"
May 29 10:35:17.705800 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:17.705785    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" path="/var/lib/kubelet/pods/af791824-7fb4-4d56-b945-a2ac38b040c3/volumes"
May 29 10:35:56.246699 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.246661    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"]
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247053    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="online"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247066    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="online"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247078    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="feast-init"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247089    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="feast-init"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247104    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="registry"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247109    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="registry"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247116    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="offline"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247120    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="offline"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247130    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="ui"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247135    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="ui"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247142    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="feast-init"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247146    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="feast-init"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247156    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="offline"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247164    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="offline"
May 29 10:35:56.247167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247173    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="ui"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247178    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="ui"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247189    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="online"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247194    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="online"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247201    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="feast-apply"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247206    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="feast-apply"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247212    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="feast-apply"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247216    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="feast-apply"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247267    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="offline"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247278    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="registry"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247284    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="online"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247291    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="ui"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247296    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="online"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247303    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="af791824-7fb4-4d56-b945-a2ac38b040c3" containerName="offline"
May 29 10:35:56.247659 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.247309    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8b28b5a0-4228-4e8c-965f-984a3b683d87" containerName="ui"
May 29 10:35:56.250433 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.250415    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.252699 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.252674    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 29 10:35:56.252699 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.252689    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 29 10:35:56.252873 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.252672    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 29 10:35:56.252972 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.252957    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 29 10:35:56.253028 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.252986    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 10:35:56.253079 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.252986    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 10:35:56.253375 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.253359    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-jz6ql\""
May 29 10:35:56.258702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.258684    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"]
May 29 10:35:56.350491 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.350451    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsvfb\" (UniqueName: \"kubernetes.io/projected/8a0d3aae-36c5-4dcc-a050-89387748b390-kube-api-access-nsvfb\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.350702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.350508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.350702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.350545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.350702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.350577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.350702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.350592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d3aae-36c5-4dcc-a050-89387748b390-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.350702 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.350664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.451164 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.451125    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.451164 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.451167    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d3aae-36c5-4dcc-a050-89387748b390-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.451411 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.451217    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.451411 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.451279    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nsvfb\" (UniqueName: \"kubernetes.io/projected/8a0d3aae-36c5-4dcc-a050-89387748b390-kube-api-access-nsvfb\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.451411 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.451327    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.451411 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.451350    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.451663 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:35:56.451441    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 29 10:35:56.451663 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:35:56.451450    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 29 10:35:56.451663 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:35:56.451518    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-ui-tls podName:8a0d3aae-36c5-4dcc-a050-89387748b390 nodeName:}" failed. No retries permitted until 2026-05-29 10:35:56.951494854 +0000 UTC m=+1379.848054453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-6x9rc" (UID: "8a0d3aae-36c5-4dcc-a050-89387748b390") : secret "feast-simple-feast-setup-ui-tls" not found
May 29 10:35:56.451663 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:35:56.451537    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-offline-tls podName:8a0d3aae-36c5-4dcc-a050-89387748b390 nodeName:}" failed. No retries permitted until 2026-05-29 10:35:56.951527359 +0000 UTC m=+1379.848086963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-6x9rc" (UID: "8a0d3aae-36c5-4dcc-a050-89387748b390") : secret "feast-simple-feast-setup-offline-tls" not found
May 29 10:35:56.451871 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.451681    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d3aae-36c5-4dcc-a050-89387748b390-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.453915 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.453896    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.453965 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.453896    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.462540 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.462512    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsvfb\" (UniqueName: \"kubernetes.io/projected/8a0d3aae-36c5-4dcc-a050-89387748b390-kube-api-access-nsvfb\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.955480 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.955444    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.955480 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.955482    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.957925 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.957901    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:56.957925 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:56.957921    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6x9rc\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:57.172455 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:57.172421    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:35:57.304008 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:57.303979    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"]
May 29 10:35:57.305957 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:35:57.305927    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a0d3aae_36c5_4dcc_a050_89387748b390.slice/crio-d1b9f572661f23713c8828f739a2d895712532e9b7092228c1595fcc5f053b21 WatchSource:0}: Error finding container d1b9f572661f23713c8828f739a2d895712532e9b7092228c1595fcc5f053b21: Status 404 returned error can't find the container with id d1b9f572661f23713c8828f739a2d895712532e9b7092228c1595fcc5f053b21
May 29 10:35:57.307661 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:57.307628    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 10:35:57.373677 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:57.373630    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerStarted","Data":"d1b9f572661f23713c8828f739a2d895712532e9b7092228c1595fcc5f053b21"}
May 29 10:35:58.377984 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:35:58.377944    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerStarted","Data":"6ef46874823bb71a827d77f438d4d862bcedb62ab6ba4a55b14521bca031d082"}
May 29 10:36:02.391012 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:02.390977    2562 generic.go:358] "Generic (PLEG): container finished" podID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerID="6ef46874823bb71a827d77f438d4d862bcedb62ab6ba4a55b14521bca031d082" exitCode=0
May 29 10:36:02.391426 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:02.391065    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerDied","Data":"6ef46874823bb71a827d77f438d4d862bcedb62ab6ba4a55b14521bca031d082"}
May 29 10:36:03.395983 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:03.395950    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerStarted","Data":"5d9c402b1396ee0cee571d1e4ae24c36acdfc90fc1c3d44da49745bb45d8071a"}
May 29 10:36:08.411302 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:08.411268    2562 generic.go:358] "Generic (PLEG): container finished" podID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerID="5d9c402b1396ee0cee571d1e4ae24c36acdfc90fc1c3d44da49745bb45d8071a" exitCode=0
May 29 10:36:08.411780 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:08.411331    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerDied","Data":"5d9c402b1396ee0cee571d1e4ae24c36acdfc90fc1c3d44da49745bb45d8071a"}
May 29 10:36:09.419749 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:09.419708    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerStarted","Data":"816342b4a820f011e57008ac28a44a267cd2b5bbaddf43468f57d2b3531241c7"}
May 29 10:36:09.420258 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:09.419763    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerStarted","Data":"c6a102fcd6a4bd6d391926de5425b46b037d81b1b275bcbd67dfd39f1b2a68ab"}
May 29 10:36:09.420258 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:09.419780    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerStarted","Data":"117b0328f1612dedde3f6a87da446c04e9d7ea5066f433f04784f0ae91fe8bfb"}
May 29 10:36:09.420258 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:09.419794    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerStarted","Data":"c22c8af1fb9a3d3679f150130810f0017ba2460ae090f47d961a1b6f4bdae15b"}
May 29 10:36:09.442256 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:09.442194    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podStartSLOduration=13.442174041 podStartE2EDuration="13.442174041s" podCreationTimestamp="2026-05-29 10:35:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:36:09.44144209 +0000 UTC m=+1392.338001711" watchObservedRunningTime="2026-05-29 10:36:09.442174041 +0000 UTC m=+1392.338733660"
May 29 10:36:12.172804 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:12.172755    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:12.172804 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:12.172806    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:12.173360 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:12.172823    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:12.173360 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:12.172836    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:12.174724 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:12.174687    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.34:8816: connect: connection refused"
May 29 10:36:12.174842 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:12.174718    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused"
May 29 10:36:12.174842 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:12.174743    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.34:8443: connect: connection refused"
May 29 10:36:12.175057 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:12.175013    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused"
May 29 10:36:15.174306 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.174274    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.174723 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.174395    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.174723 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.174602    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.174939 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.174878    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.174939 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.174908    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.174939 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.174921    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.175253 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.175235    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.175347 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.175336    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.175380 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.175372    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.178688 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.178673    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.441302 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.441215    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:36:15.444242 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:36:15.444217    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:37:17.179244 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.179214    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"]
May 29 10:37:17.179721 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.179527    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="registry" containerID="cri-o://c22c8af1fb9a3d3679f150130810f0017ba2460ae090f47d961a1b6f4bdae15b" gracePeriod=30
May 29 10:37:17.179721 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.179582    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="offline" containerID="cri-o://c6a102fcd6a4bd6d391926de5425b46b037d81b1b275bcbd67dfd39f1b2a68ab" gracePeriod=30
May 29 10:37:17.179721 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.179604    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="online" containerID="cri-o://117b0328f1612dedde3f6a87da446c04e9d7ea5066f433f04784f0ae91fe8bfb" gracePeriod=30
May 29 10:37:17.179721 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.179672    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="ui" containerID="cri-o://816342b4a820f011e57008ac28a44a267cd2b5bbaddf43468f57d2b3531241c7" gracePeriod=30
May 29 10:37:17.464476 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.464387    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"]
May 29 10:37:17.469335 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.469309    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.471415 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.471395    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-nqpp5\""
May 29 10:37:17.478552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.478522    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"]
May 29 10:37:17.577154 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.577115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.577154 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.577150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7417b6c2-8fec-4720-95d5-c31990580571-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.577416 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.577188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.577416 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.577215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.577416 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.577246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.577416 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.577323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmq4\" (UniqueName: \"kubernetes.io/projected/7417b6c2-8fec-4720-95d5-c31990580571-kube-api-access-bwmq4\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.678086 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.678048    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.678251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.678129    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.678251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.678162    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmq4\" (UniqueName: \"kubernetes.io/projected/7417b6c2-8fec-4720-95d5-c31990580571-kube-api-access-bwmq4\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.678251 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:37:17.678211    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 29 10:37:17.678251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.678233    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.678455 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.678259    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7417b6c2-8fec-4720-95d5-c31990580571-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.678455 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:37:17.678264    2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 29 10:37:17.678455 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:37:17.678305    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-online-tls podName:7417b6c2-8fec-4720-95d5-c31990580571 nodeName:}" failed. No retries permitted until 2026-05-29 10:37:18.178281822 +0000 UTC m=+1461.074841425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-online-tls") pod "feast-simple-feast-setup-664fcfcc58-vtgq7" (UID: "7417b6c2-8fec-4720-95d5-c31990580571") : secret "feast-simple-feast-setup-online-tls" not found
May 29 10:37:17.678455 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:37:17.678340    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-ui-tls podName:7417b6c2-8fec-4720-95d5-c31990580571 nodeName:}" failed. No retries permitted until 2026-05-29 10:37:18.178321938 +0000 UTC m=+1461.074881538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-vtgq7" (UID: "7417b6c2-8fec-4720-95d5-c31990580571") : secret "feast-simple-feast-setup-ui-tls" not found
May 29 10:37:17.678455 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.678401    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.678751 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.678719    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7417b6c2-8fec-4720-95d5-c31990580571-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.681031 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.681012    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.681111 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.681068    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:17.689593 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:17.689565    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmq4\" (UniqueName: \"kubernetes.io/projected/7417b6c2-8fec-4720-95d5-c31990580571-kube-api-access-bwmq4\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:18.182185 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.182092    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:18.182185 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.182156    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:18.184763 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.184732    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:18.184899 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.184747    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-vtgq7\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:18.380577 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.380518    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:18.515867 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.515721    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"]
May 29 10:37:18.517968 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:37:18.517942    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7417b6c2_8fec_4720_95d5_c31990580571.slice/crio-21628b90e3b660d5973b6db58c6f456802cc507763e05548dddcfc1d743efb41 WatchSource:0}: Error finding container 21628b90e3b660d5973b6db58c6f456802cc507763e05548dddcfc1d743efb41: Status 404 returned error can't find the container with id 21628b90e3b660d5973b6db58c6f456802cc507763e05548dddcfc1d743efb41
May 29 10:37:18.630907 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.630873    2562 generic.go:358] "Generic (PLEG): container finished" podID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerID="816342b4a820f011e57008ac28a44a267cd2b5bbaddf43468f57d2b3531241c7" exitCode=0
May 29 10:37:18.630907 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.630899    2562 generic.go:358] "Generic (PLEG): container finished" podID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerID="117b0328f1612dedde3f6a87da446c04e9d7ea5066f433f04784f0ae91fe8bfb" exitCode=0
May 29 10:37:18.631122 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.630950    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerDied","Data":"816342b4a820f011e57008ac28a44a267cd2b5bbaddf43468f57d2b3531241c7"}
May 29 10:37:18.631122 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.630980    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerDied","Data":"117b0328f1612dedde3f6a87da446c04e9d7ea5066f433f04784f0ae91fe8bfb"}
May 29 10:37:18.632210 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.632183    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerStarted","Data":"c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c"}
May 29 10:37:18.632331 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:18.632218    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerStarted","Data":"21628b90e3b660d5973b6db58c6f456802cc507763e05548dddcfc1d743efb41"}
May 29 10:37:23.650240 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:23.650205    2562 generic.go:358] "Generic (PLEG): container finished" podID="7417b6c2-8fec-4720-95d5-c31990580571" containerID="c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c" exitCode=0
May 29 10:37:23.650627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:23.650279    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerDied","Data":"c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c"}
May 29 10:37:24.656432 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:24.656396    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerStarted","Data":"da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572"}
May 29 10:37:25.175331 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:25.175289    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.34:8443: connect: connection refused"
May 29 10:37:25.442130 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:25.442041    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused"
May 29 10:37:29.672172 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:29.672136    2562 generic.go:358] "Generic (PLEG): container finished" podID="7417b6c2-8fec-4720-95d5-c31990580571" containerID="da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572" exitCode=0
May 29 10:37:29.672576 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:29.672207    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerDied","Data":"da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572"}
May 29 10:37:30.679817 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:30.679778    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerStarted","Data":"619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216"}
May 29 10:37:30.680276 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:30.679829    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerStarted","Data":"910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622"}
May 29 10:37:30.680276 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:30.679846    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerStarted","Data":"c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8"}
May 29 10:37:30.680276 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:30.679860    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerStarted","Data":"521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa"}
May 29 10:37:30.703438 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:30.703375    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podStartSLOduration=13.703354917 podStartE2EDuration="13.703354917s" podCreationTimestamp="2026-05-29 10:37:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:37:30.701811094 +0000 UTC m=+1473.598370727" watchObservedRunningTime="2026-05-29 10:37:30.703354917 +0000 UTC m=+1473.599914537"
May 29 10:37:33.380877 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:33.380830    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:33.381444 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:33.381421    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:33.381444 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:33.381445    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:33.381675 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:33.381457    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:33.383011 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:33.382980    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused"
May 29 10:37:33.383230 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:33.382995    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.35:8816: connect: connection refused"
May 29 10:37:33.383316 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:33.383005    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused"
May 29 10:37:33.383316 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:33.383058    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 29 10:37:35.175822 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:35.175601    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.34:8443: connect: connection refused"
May 29 10:37:35.441782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:35.441685    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused"
May 29 10:37:36.382424 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.382394    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.382890 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.382494    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.382890 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.382799    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.383412 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.383314    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.383412 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.383342    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.383412 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.383355    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.383603 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.383571    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.383686 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.383618    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.383752 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.383738    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.386798 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.386784    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.704854 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.704771    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:36.707726 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:36.707705    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:37:45.175912 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:45.175865    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.34:8443: connect: connection refused"
May 29 10:37:45.176367 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:45.176023    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:37:45.441699 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:45.441604    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused"
May 29 10:37:45.441852 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:45.441765    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:37:47.746938 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.746905    2562 generic.go:358] "Generic (PLEG): container finished" podID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerID="c6a102fcd6a4bd6d391926de5425b46b037d81b1b275bcbd67dfd39f1b2a68ab" exitCode=137
May 29 10:37:47.746938 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.746932    2562 generic.go:358] "Generic (PLEG): container finished" podID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerID="c22c8af1fb9a3d3679f150130810f0017ba2460ae090f47d961a1b6f4bdae15b" exitCode=137
May 29 10:37:47.747313 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.746976    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerDied","Data":"c6a102fcd6a4bd6d391926de5425b46b037d81b1b275bcbd67dfd39f1b2a68ab"}
May 29 10:37:47.747313 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.747009    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerDied","Data":"c22c8af1fb9a3d3679f150130810f0017ba2460ae090f47d961a1b6f4bdae15b"}
May 29 10:37:47.831298 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.831274    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:37:47.956552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.956475    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-offline-tls\") pod \"8a0d3aae-36c5-4dcc-a050-89387748b390\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") "
May 29 10:37:47.956552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.956511    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-online-tls\") pod \"8a0d3aae-36c5-4dcc-a050-89387748b390\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") "
May 29 10:37:47.956552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.956531    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-ui-tls\") pod \"8a0d3aae-36c5-4dcc-a050-89387748b390\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") "
May 29 10:37:47.956839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.956562    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-registry-tls\") pod \"8a0d3aae-36c5-4dcc-a050-89387748b390\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") "
May 29 10:37:47.956839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.956610    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsvfb\" (UniqueName: \"kubernetes.io/projected/8a0d3aae-36c5-4dcc-a050-89387748b390-kube-api-access-nsvfb\") pod \"8a0d3aae-36c5-4dcc-a050-89387748b390\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") "
May 29 10:37:47.956839 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.956738    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d3aae-36c5-4dcc-a050-89387748b390-feast-data\") pod \"8a0d3aae-36c5-4dcc-a050-89387748b390\" (UID: \"8a0d3aae-36c5-4dcc-a050-89387748b390\") "
May 29 10:37:47.957552 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.957483    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0d3aae-36c5-4dcc-a050-89387748b390-feast-data" (OuterVolumeSpecName: "feast-data") pod "8a0d3aae-36c5-4dcc-a050-89387748b390" (UID: "8a0d3aae-36c5-4dcc-a050-89387748b390"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:37:47.959097 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.959064    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "8a0d3aae-36c5-4dcc-a050-89387748b390" (UID: "8a0d3aae-36c5-4dcc-a050-89387748b390"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:37:47.959215 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.959103    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-online-tls" (OuterVolumeSpecName: "online-tls") pod "8a0d3aae-36c5-4dcc-a050-89387748b390" (UID: "8a0d3aae-36c5-4dcc-a050-89387748b390"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:37:47.959215 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.959107    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0d3aae-36c5-4dcc-a050-89387748b390-kube-api-access-nsvfb" (OuterVolumeSpecName: "kube-api-access-nsvfb") pod "8a0d3aae-36c5-4dcc-a050-89387748b390" (UID: "8a0d3aae-36c5-4dcc-a050-89387748b390"). InnerVolumeSpecName "kube-api-access-nsvfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:37:47.959370 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.959219    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "8a0d3aae-36c5-4dcc-a050-89387748b390" (UID: "8a0d3aae-36c5-4dcc-a050-89387748b390"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:37:47.959370 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:47.959242    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8a0d3aae-36c5-4dcc-a050-89387748b390" (UID: "8a0d3aae-36c5-4dcc-a050-89387748b390"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:37:48.057774 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.057735    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsvfb\" (UniqueName: \"kubernetes.io/projected/8a0d3aae-36c5-4dcc-a050-89387748b390-kube-api-access-nsvfb\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:37:48.057774 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.057770    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0d3aae-36c5-4dcc-a050-89387748b390-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:37:48.057774 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.057779    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:37:48.058004 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.057787    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:37:48.058004 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.057797    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-ui-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:37:48.058004 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.057805    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a0d3aae-36c5-4dcc-a050-89387748b390-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:37:48.752030 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.751999    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc" event={"ID":"8a0d3aae-36c5-4dcc-a050-89387748b390","Type":"ContainerDied","Data":"d1b9f572661f23713c8828f739a2d895712532e9b7092228c1595fcc5f053b21"}
May 29 10:37:48.752461 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.752042    2562 scope.go:117] "RemoveContainer" containerID="816342b4a820f011e57008ac28a44a267cd2b5bbaddf43468f57d2b3531241c7"
May 29 10:37:48.752461 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.752005    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"
May 29 10:37:48.760715 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.760700    2562 scope.go:117] "RemoveContainer" containerID="c6a102fcd6a4bd6d391926de5425b46b037d81b1b275bcbd67dfd39f1b2a68ab"
May 29 10:37:48.767776 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.767757    2562 scope.go:117] "RemoveContainer" containerID="117b0328f1612dedde3f6a87da446c04e9d7ea5066f433f04784f0ae91fe8bfb"
May 29 10:37:48.772670 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.772620    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"]
May 29 10:37:48.775251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.775232    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6x9rc"]
May 29 10:37:48.776801 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.776784    2562 scope.go:117] "RemoveContainer" containerID="c22c8af1fb9a3d3679f150130810f0017ba2460ae090f47d961a1b6f4bdae15b"
May 29 10:37:48.785857 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.785830    2562 scope.go:117] "RemoveContainer" containerID="5d9c402b1396ee0cee571d1e4ae24c36acdfc90fc1c3d44da49745bb45d8071a"
May 29 10:37:48.793336 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:48.793319    2562 scope.go:117] "RemoveContainer" containerID="6ef46874823bb71a827d77f438d4d862bcedb62ab6ba4a55b14521bca031d082"
May 29 10:37:49.703486 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:49.703456    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" path="/var/lib/kubelet/pods/8a0d3aae-36c5-4dcc-a050-89387748b390/volumes"
May 29 10:37:57.698310 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:57.698278    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:37:57.701798 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:37:57.701775    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:38:39.308368 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308332    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"]
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308712    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="ui"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308725    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="ui"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308739    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="online"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308745    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="online"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308758    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="feast-apply"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308763    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="feast-apply"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308770    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="offline"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308774    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="offline"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308785    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="feast-init"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308790    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="feast-init"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308800    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="registry"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308805    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="registry"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308854    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="ui"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308862    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="online"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308870    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="registry"
May 29 10:38:39.308872 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.308877    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8a0d3aae-36c5-4dcc-a050-89387748b390" containerName="offline"
May 29 10:38:39.312334 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.312319    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.314369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.314343    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 29 10:38:39.314503 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.314387    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 29 10:38:39.314503 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.314402    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 29 10:38:39.314503 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.314402    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 29 10:38:39.314503 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.314485    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-rtqxc\""
May 29 10:38:39.314503 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.314387    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 29 10:38:39.314846 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.314825    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 29 10:38:39.320610 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.320589    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"]
May 29 10:38:39.383965 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.383926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a333189-ccce-45de-83c9-8d489b883ff7-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.383965 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.383968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.384229 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.383995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a333189-ccce-45de-83c9-8d489b883ff7-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.384229 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.384074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.384229 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.384111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7hk\" (UniqueName: \"kubernetes.io/projected/2a333189-ccce-45de-83c9-8d489b883ff7-kube-api-access-gp7hk\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.384229 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.384166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.484590 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.484546    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a333189-ccce-45de-83c9-8d489b883ff7-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.484804 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.484603    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.484804 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.484680    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a333189-ccce-45de-83c9-8d489b883ff7-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.484804 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.484729    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.484804 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.484760    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7hk\" (UniqueName: \"kubernetes.io/projected/2a333189-ccce-45de-83c9-8d489b883ff7-kube-api-access-gp7hk\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.484804 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.484796    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.485081 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:38:39.484937    2562 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 10:38:39.485081 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:38:39.484995    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-online-tls podName:2a333189-ccce-45de-83c9-8d489b883ff7 nodeName:}" failed. No retries permitted until 2026-05-29 10:38:39.984973848 +0000 UTC m=+1542.881533446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-online-tls") pod "feast-simple-feast-remote-setup-5cff94746b-jdvfv" (UID: "2a333189-ccce-45de-83c9-8d489b883ff7") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 10:38:39.485234 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.485210    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a333189-ccce-45de-83c9-8d489b883ff7-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.485379 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.485360    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a333189-ccce-45de-83c9-8d489b883ff7-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.487248 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.487225    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.487418 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.487402    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.496475 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.496448    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7hk\" (UniqueName: \"kubernetes.io/projected/2a333189-ccce-45de-83c9-8d489b883ff7-kube-api-access-gp7hk\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.990139 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.990079    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:39.992528 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:39.992508    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-jdvfv\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:40.222887 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:40.222846    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:40.353356 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:40.353331    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"]
May 29 10:38:40.355103 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:38:40.355077    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a333189_ccce_45de_83c9_8d489b883ff7.slice/crio-98930882a9da2947197e203a27fe8366b1e176bf2a298f07beea67b7105457c9 WatchSource:0}: Error finding container 98930882a9da2947197e203a27fe8366b1e176bf2a298f07beea67b7105457c9: Status 404 returned error can't find the container with id 98930882a9da2947197e203a27fe8366b1e176bf2a298f07beea67b7105457c9
May 29 10:38:40.906624 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:40.906580    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerStarted","Data":"8cfe0ddc2d72e593dd8bfecd158b34646ba5a18e7c42214366da62830f330dd5"}
May 29 10:38:40.906819 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:40.906658    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerStarted","Data":"98930882a9da2947197e203a27fe8366b1e176bf2a298f07beea67b7105457c9"}
May 29 10:38:45.925908 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:45.925875    2562 generic.go:358] "Generic (PLEG): container finished" podID="2a333189-ccce-45de-83c9-8d489b883ff7" containerID="8cfe0ddc2d72e593dd8bfecd158b34646ba5a18e7c42214366da62830f330dd5" exitCode=0
May 29 10:38:45.926275 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:45.925914    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerDied","Data":"8cfe0ddc2d72e593dd8bfecd158b34646ba5a18e7c42214366da62830f330dd5"}
May 29 10:38:46.930745 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:46.930707    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerStarted","Data":"cd195c5ca03e2852e49c71abcadbe0e3fde3aabb04388f9ccbef11e5d82d779b"}
May 29 10:38:51.947108 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:51.947021    2562 generic.go:358] "Generic (PLEG): container finished" podID="2a333189-ccce-45de-83c9-8d489b883ff7" containerID="cd195c5ca03e2852e49c71abcadbe0e3fde3aabb04388f9ccbef11e5d82d779b" exitCode=0
May 29 10:38:51.947108 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:51.947097    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerDied","Data":"cd195c5ca03e2852e49c71abcadbe0e3fde3aabb04388f9ccbef11e5d82d779b"}
May 29 10:38:52.953231 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:52.953189    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerStarted","Data":"42be88a6fdfd0cf7eda2937f6b793416193503940611e4b7360428b60eecdac0"}
May 29 10:38:52.953606 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:52.953247    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerStarted","Data":"fb97a4ae73ab665d94508757e6d6989dd227aab1dab6a41abf281f8b4f2f56aa"}
May 29 10:38:52.953606 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:52.953264    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerStarted","Data":"14b8fece2c7bceba5e0dae583e1496cbca597e8574d8573046f42498c41edc85"}
May 29 10:38:52.977136 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:52.977076    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podStartSLOduration=13.977055431 podStartE2EDuration="13.977055431s" podCreationTimestamp="2026-05-29 10:38:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:38:52.975753399 +0000 UTC m=+1555.872313019" watchObservedRunningTime="2026-05-29 10:38:52.977055431 +0000 UTC m=+1555.873615051"
May 29 10:38:55.224078 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:55.224041    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:55.224534 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:55.224092    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:55.224534 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:55.224107    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:55.225721 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:55.225689    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.36:8816: connect: connection refused"
May 29 10:38:55.225853 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:55.225718    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused"
May 29 10:38:55.225853 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:55.225737    2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 29 10:38:58.224319 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:58.224288    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:58.224803 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:58.224504    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:58.225044 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:58.225023    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:58.225131 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:58.225056    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:58.225231 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:58.225211    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:58.225362 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:58.225347    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:58.228451 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:58.228433    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:58.973692 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:58.973654    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:38:58.976761 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:38:58.976736    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:40:01.117046 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:01.117008    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"]
May 29 10:40:01.117591 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:01.117406    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="online" containerID="cri-o://14b8fece2c7bceba5e0dae583e1496cbca597e8574d8573046f42498c41edc85" gracePeriod=30
May 29 10:40:01.117591 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:01.117461    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="ui" containerID="cri-o://42be88a6fdfd0cf7eda2937f6b793416193503940611e4b7360428b60eecdac0" gracePeriod=30
May 29 10:40:01.117591 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:01.117492    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="offline" containerID="cri-o://fb97a4ae73ab665d94508757e6d6989dd227aab1dab6a41abf281f8b4f2f56aa" gracePeriod=30
May 29 10:40:01.259074 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:01.259027    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"]
May 29 10:40:01.259533 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:01.259369    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="registry" containerID="cri-o://521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa" gracePeriod=30
May 29 10:40:01.259533 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:01.259440    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="ui" containerID="cri-o://619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216" gracePeriod=30
May 29 10:40:01.259533 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:01.259428    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="offline" containerID="cri-o://910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622" gracePeriod=30
May 29 10:40:01.259533 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:01.259439    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="online" containerID="cri-o://c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8" gracePeriod=30
May 29 10:40:02.168826 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:02.168729    2562 generic.go:358] "Generic (PLEG): container finished" podID="7417b6c2-8fec-4720-95d5-c31990580571" containerID="619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216" exitCode=0
May 29 10:40:02.168826 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:02.168761    2562 generic.go:358] "Generic (PLEG): container finished" podID="7417b6c2-8fec-4720-95d5-c31990580571" containerID="c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8" exitCode=0
May 29 10:40:02.168826 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:02.168759    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerDied","Data":"619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216"}
May 29 10:40:02.168826 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:02.168798    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerDied","Data":"c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8"}
May 29 10:40:02.171080 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:02.171045    2562 generic.go:358] "Generic (PLEG): container finished" podID="2a333189-ccce-45de-83c9-8d489b883ff7" containerID="42be88a6fdfd0cf7eda2937f6b793416193503940611e4b7360428b60eecdac0" exitCode=0
May 29 10:40:02.171080 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:02.171064    2562 generic.go:358] "Generic (PLEG): container finished" podID="2a333189-ccce-45de-83c9-8d489b883ff7" containerID="14b8fece2c7bceba5e0dae583e1496cbca597e8574d8573046f42498c41edc85" exitCode=0
May 29 10:40:02.171080 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:02.171072    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerDied","Data":"42be88a6fdfd0cf7eda2937f6b793416193503940611e4b7360428b60eecdac0"}
May 29 10:40:02.171266 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:02.171098    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerDied","Data":"14b8fece2c7bceba5e0dae583e1496cbca597e8574d8573046f42498c41edc85"}
May 29 10:40:06.383977 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:06.383929    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused"
May 29 10:40:06.705734 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:06.705617    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 29 10:40:08.225781 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:08.225727    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused"
May 29 10:40:08.974003 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:08.973956    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 29 10:40:16.383947 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:16.383886    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused"
May 29 10:40:16.705198 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:16.705106    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 29 10:40:18.225882 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:18.225835    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused"
May 29 10:40:18.974609 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:18.974566    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 29 10:40:26.384320 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:26.384271    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused"
May 29 10:40:26.384769 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:26.384418    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:40:26.705010 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:26.704913    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 29 10:40:26.705182 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:26.705042    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:40:28.225224 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:28.225178    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused"
May 29 10:40:28.225706 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:28.225313    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:40:28.973844 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:28.973804    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 29 10:40:28.974033 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:28.973910    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:40:31.259516 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.259473    2562 generic.go:358] "Generic (PLEG): container finished" podID="2a333189-ccce-45de-83c9-8d489b883ff7" containerID="fb97a4ae73ab665d94508757e6d6989dd227aab1dab6a41abf281f8b4f2f56aa" exitCode=137
May 29 10:40:31.259877 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.259536    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerDied","Data":"fb97a4ae73ab665d94508757e6d6989dd227aab1dab6a41abf281f8b4f2f56aa"}
May 29 10:40:31.770514 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.770490    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:40:31.793855 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.793821    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a333189-ccce-45de-83c9-8d489b883ff7-registry-tls\") pod \"2a333189-ccce-45de-83c9-8d489b883ff7\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") "
May 29 10:40:31.794034 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.793876    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp7hk\" (UniqueName: \"kubernetes.io/projected/2a333189-ccce-45de-83c9-8d489b883ff7-kube-api-access-gp7hk\") pod \"2a333189-ccce-45de-83c9-8d489b883ff7\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") "
May 29 10:40:31.794034 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.793940    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-online-tls\") pod \"2a333189-ccce-45de-83c9-8d489b883ff7\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") "
May 29 10:40:31.794034 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.793988    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a333189-ccce-45de-83c9-8d489b883ff7-feast-data\") pod \"2a333189-ccce-45de-83c9-8d489b883ff7\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") "
May 29 10:40:31.794034 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.794020    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-offline-tls\") pod \"2a333189-ccce-45de-83c9-8d489b883ff7\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") "
May 29 10:40:31.794271 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.794069    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-ui-tls\") pod \"2a333189-ccce-45de-83c9-8d489b883ff7\" (UID: \"2a333189-ccce-45de-83c9-8d489b883ff7\") "
May 29 10:40:31.794662 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.794586    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a333189-ccce-45de-83c9-8d489b883ff7-feast-data" (OuterVolumeSpecName: "feast-data") pod "2a333189-ccce-45de-83c9-8d489b883ff7" (UID: "2a333189-ccce-45de-83c9-8d489b883ff7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:40:31.794755 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.794657    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a333189-ccce-45de-83c9-8d489b883ff7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2a333189-ccce-45de-83c9-8d489b883ff7" (UID: "2a333189-ccce-45de-83c9-8d489b883ff7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:40:31.797274 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.797243    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2a333189-ccce-45de-83c9-8d489b883ff7" (UID: "2a333189-ccce-45de-83c9-8d489b883ff7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:40:31.797404 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.797380    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-online-tls" (OuterVolumeSpecName: "online-tls") pod "2a333189-ccce-45de-83c9-8d489b883ff7" (UID: "2a333189-ccce-45de-83c9-8d489b883ff7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:40:31.797539 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.797507    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2a333189-ccce-45de-83c9-8d489b883ff7" (UID: "2a333189-ccce-45de-83c9-8d489b883ff7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:40:31.798414 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.798392    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a333189-ccce-45de-83c9-8d489b883ff7-kube-api-access-gp7hk" (OuterVolumeSpecName: "kube-api-access-gp7hk") pod "2a333189-ccce-45de-83c9-8d489b883ff7" (UID: "2a333189-ccce-45de-83c9-8d489b883ff7"). InnerVolumeSpecName "kube-api-access-gp7hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:40:31.895174 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.895146    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:31.895174 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.895172    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a333189-ccce-45de-83c9-8d489b883ff7-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:31.895174 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.895180    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:31.895174 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.895190    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a333189-ccce-45de-83c9-8d489b883ff7-ui-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:31.895448 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.895198    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a333189-ccce-45de-83c9-8d489b883ff7-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:31.895448 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.895206    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp7hk\" (UniqueName: \"kubernetes.io/projected/2a333189-ccce-45de-83c9-8d489b883ff7-kube-api-access-gp7hk\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:31.899310 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.899281    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:40:31.995502 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.995457    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-registry-tls\") pod \"7417b6c2-8fec-4720-95d5-c31990580571\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") "
May 29 10:40:31.995502 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.995512    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwmq4\" (UniqueName: \"kubernetes.io/projected/7417b6c2-8fec-4720-95d5-c31990580571-kube-api-access-bwmq4\") pod \"7417b6c2-8fec-4720-95d5-c31990580571\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") "
May 29 10:40:31.995778 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.995539    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-online-tls\") pod \"7417b6c2-8fec-4720-95d5-c31990580571\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") "
May 29 10:40:31.995778 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.995570    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-offline-tls\") pod \"7417b6c2-8fec-4720-95d5-c31990580571\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") "
May 29 10:40:31.995778 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.995609    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-ui-tls\") pod \"7417b6c2-8fec-4720-95d5-c31990580571\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") "
May 29 10:40:31.995778 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.995700    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7417b6c2-8fec-4720-95d5-c31990580571-feast-data\") pod \"7417b6c2-8fec-4720-95d5-c31990580571\" (UID: \"7417b6c2-8fec-4720-95d5-c31990580571\") "
May 29 10:40:31.996542 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.996510    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7417b6c2-8fec-4720-95d5-c31990580571-feast-data" (OuterVolumeSpecName: "feast-data") pod "7417b6c2-8fec-4720-95d5-c31990580571" (UID: "7417b6c2-8fec-4720-95d5-c31990580571"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:40:31.997928 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.997906    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "7417b6c2-8fec-4720-95d5-c31990580571" (UID: "7417b6c2-8fec-4720-95d5-c31990580571"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:40:31.998301 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.998277    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7417b6c2-8fec-4720-95d5-c31990580571" (UID: "7417b6c2-8fec-4720-95d5-c31990580571"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:40:31.998301 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.998290    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7417b6c2-8fec-4720-95d5-c31990580571-kube-api-access-bwmq4" (OuterVolumeSpecName: "kube-api-access-bwmq4") pod "7417b6c2-8fec-4720-95d5-c31990580571" (UID: "7417b6c2-8fec-4720-95d5-c31990580571"). InnerVolumeSpecName "kube-api-access-bwmq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:40:31.998431 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.998298    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7417b6c2-8fec-4720-95d5-c31990580571" (UID: "7417b6c2-8fec-4720-95d5-c31990580571"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:40:31.998431 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:31.998312    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-online-tls" (OuterVolumeSpecName: "online-tls") pod "7417b6c2-8fec-4720-95d5-c31990580571" (UID: "7417b6c2-8fec-4720-95d5-c31990580571"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:40:32.096419 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.096392    2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7417b6c2-8fec-4720-95d5-c31990580571-feast-data\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:32.096419 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.096417    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-registry-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:32.096419 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.096426    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwmq4\" (UniqueName: \"kubernetes.io/projected/7417b6c2-8fec-4720-95d5-c31990580571-kube-api-access-bwmq4\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:32.096680 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.096436    2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-online-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:32.096680 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.096451    2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-offline-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:32.096680 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.096462    2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7417b6c2-8fec-4720-95d5-c31990580571-ui-tls\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:40:32.264610 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.264519    2562 generic.go:358] "Generic (PLEG): container finished" podID="7417b6c2-8fec-4720-95d5-c31990580571" containerID="910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622" exitCode=137
May 29 10:40:32.264610 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.264545    2562 generic.go:358] "Generic (PLEG): container finished" podID="7417b6c2-8fec-4720-95d5-c31990580571" containerID="521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa" exitCode=137
May 29 10:40:32.264610 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.264587    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"
May 29 10:40:32.264610 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.264589    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerDied","Data":"910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622"}
May 29 10:40:32.265230 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.264623    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerDied","Data":"521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa"}
May 29 10:40:32.265230 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.264671    2562 scope.go:117] "RemoveContainer" containerID="619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216"
May 29 10:40:32.265230 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.264674    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7" event={"ID":"7417b6c2-8fec-4720-95d5-c31990580571","Type":"ContainerDied","Data":"21628b90e3b660d5973b6db58c6f456802cc507763e05548dddcfc1d743efb41"}
May 29 10:40:32.267024 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.267001    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv" event={"ID":"2a333189-ccce-45de-83c9-8d489b883ff7","Type":"ContainerDied","Data":"98930882a9da2947197e203a27fe8366b1e176bf2a298f07beea67b7105457c9"}
May 29 10:40:32.267147 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.267063    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"
May 29 10:40:32.274985 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.274963    2562 scope.go:117] "RemoveContainer" containerID="910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622"
May 29 10:40:32.282397 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.282376    2562 scope.go:117] "RemoveContainer" containerID="c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8"
May 29 10:40:32.287212 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.287192    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"]
May 29 10:40:32.290443 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.290423    2562 scope.go:117] "RemoveContainer" containerID="521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa"
May 29 10:40:32.291473 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.291436    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-vtgq7"]
May 29 10:40:32.297458 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.297437    2562 scope.go:117] "RemoveContainer" containerID="da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572"
May 29 10:40:32.301201 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.301182    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"]
May 29 10:40:32.305401 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.305377    2562 scope.go:117] "RemoveContainer" containerID="c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c"
May 29 10:40:32.306768 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.306749    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-jdvfv"]
May 29 10:40:32.312672 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.312653    2562 scope.go:117] "RemoveContainer" containerID="619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216"
May 29 10:40:32.312912 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:40:32.312895    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216\": container with ID starting with 619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216 not found: ID does not exist" containerID="619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216"
May 29 10:40:32.312966 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.312924    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216"} err="failed to get container status \"619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216\": rpc error: code = NotFound desc = could not find container \"619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216\": container with ID starting with 619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216 not found: ID does not exist"
May 29 10:40:32.312966 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.312944    2562 scope.go:117] "RemoveContainer" containerID="910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622"
May 29 10:40:32.313175 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:40:32.313160    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622\": container with ID starting with 910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622 not found: ID does not exist" containerID="910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622"
May 29 10:40:32.313212 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.313179    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622"} err="failed to get container status \"910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622\": rpc error: code = NotFound desc = could not find container \"910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622\": container with ID starting with 910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622 not found: ID does not exist"
May 29 10:40:32.313212 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.313194    2562 scope.go:117] "RemoveContainer" containerID="c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8"
May 29 10:40:32.313395 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:40:32.313378    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8\": container with ID starting with c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8 not found: ID does not exist" containerID="c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8"
May 29 10:40:32.313436 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.313401    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8"} err="failed to get container status \"c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8\": rpc error: code = NotFound desc = could not find container \"c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8\": container with ID starting with c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8 not found: ID does not exist"
May 29 10:40:32.313436 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.313417    2562 scope.go:117] "RemoveContainer" containerID="521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa"
May 29 10:40:32.313662 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:40:32.313621    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa\": container with ID starting with 521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa not found: ID does not exist" containerID="521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa"
May 29 10:40:32.313662 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.313652    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa"} err="failed to get container status \"521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa\": rpc error: code = NotFound desc = could not find container \"521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa\": container with ID starting with 521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa not found: ID does not exist"
May 29 10:40:32.313662 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.313664    2562 scope.go:117] "RemoveContainer" containerID="da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572"
May 29 10:40:32.313893 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:40:32.313874    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572\": container with ID starting with da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572 not found: ID does not exist" containerID="da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572"
May 29 10:40:32.313955 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.313901    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572"} err="failed to get container status \"da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572\": rpc error: code = NotFound desc = could not find container \"da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572\": container with ID starting with da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572 not found: ID does not exist"
May 29 10:40:32.313955 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.313922    2562 scope.go:117] "RemoveContainer" containerID="c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c"
May 29 10:40:32.314126 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:40:32.314109    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c\": container with ID starting with c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c not found: ID does not exist" containerID="c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c"
May 29 10:40:32.314174 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314130    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c"} err="failed to get container status \"c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c\": rpc error: code = NotFound desc = could not find container \"c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c\": container with ID starting with c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c not found: ID does not exist"
May 29 10:40:32.314174 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314143    2562 scope.go:117] "RemoveContainer" containerID="619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216"
May 29 10:40:32.314379 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314361    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216"} err="failed to get container status \"619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216\": rpc error: code = NotFound desc = could not find container \"619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216\": container with ID starting with 619297b04164f3e73bc1c4a64ce580b1aa2d387aa5f442bc7abc94d23fb85216 not found: ID does not exist"
May 29 10:40:32.314428 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314381    2562 scope.go:117] "RemoveContainer" containerID="910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622"
May 29 10:40:32.314583 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314566    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622"} err="failed to get container status \"910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622\": rpc error: code = NotFound desc = could not find container \"910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622\": container with ID starting with 910136276da64c60ed0721c0cc79dd79d3c94af81b70b157e778cf94be8fa622 not found: ID does not exist"
May 29 10:40:32.314623 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314585    2562 scope.go:117] "RemoveContainer" containerID="c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8"
May 29 10:40:32.314771 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314755    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8"} err="failed to get container status \"c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8\": rpc error: code = NotFound desc = could not find container \"c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8\": container with ID starting with c48d5f0a8276db67cf773bdabe4f4dbbe8e48f961ef5303d97c9fab54f1ea4e8 not found: ID does not exist"
May 29 10:40:32.314821 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314771    2562 scope.go:117] "RemoveContainer" containerID="521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa"
May 29 10:40:32.314985 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314968    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa"} err="failed to get container status \"521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa\": rpc error: code = NotFound desc = could not find container \"521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa\": container with ID starting with 521c94444e2c7de1a116fe2854d4c53d7efbc52c13db855b56b799da80cfebfa not found: ID does not exist"
May 29 10:40:32.314985 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.314985    2562 scope.go:117] "RemoveContainer" containerID="da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572"
May 29 10:40:32.315172 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.315157    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572"} err="failed to get container status \"da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572\": rpc error: code = NotFound desc = could not find container \"da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572\": container with ID starting with da6eceba5624c2846fd5ad90b5f70302470f077bfe32a621f758ceda67c7b572 not found: ID does not exist"
May 29 10:40:32.315219 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.315173    2562 scope.go:117] "RemoveContainer" containerID="c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c"
May 29 10:40:32.315349 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.315334    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c"} err="failed to get container status \"c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c\": rpc error: code = NotFound desc = could not find container \"c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c\": container with ID starting with c16c95f19cc50acfcad54cfd93e2e2c1607881fa7c1f4b22c43c9b38fcfbe37c not found: ID does not exist"
May 29 10:40:32.315384 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.315349    2562 scope.go:117] "RemoveContainer" containerID="42be88a6fdfd0cf7eda2937f6b793416193503940611e4b7360428b60eecdac0"
May 29 10:40:32.322371 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.322357    2562 scope.go:117] "RemoveContainer" containerID="fb97a4ae73ab665d94508757e6d6989dd227aab1dab6a41abf281f8b4f2f56aa"
May 29 10:40:32.329096 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.329079    2562 scope.go:117] "RemoveContainer" containerID="14b8fece2c7bceba5e0dae583e1496cbca597e8574d8573046f42498c41edc85"
May 29 10:40:32.335550 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.335532    2562 scope.go:117] "RemoveContainer" containerID="cd195c5ca03e2852e49c71abcadbe0e3fde3aabb04388f9ccbef11e5d82d779b"
May 29 10:40:32.341955 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:32.341942    2562 scope.go:117] "RemoveContainer" containerID="8cfe0ddc2d72e593dd8bfecd158b34646ba5a18e7c42214366da62830f330dd5"
May 29 10:40:33.703751 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:33.703720    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" path="/var/lib/kubelet/pods/2a333189-ccce-45de-83c9-8d489b883ff7/volumes"
May 29 10:40:33.704257 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:33.704243    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7417b6c2-8fec-4720-95d5-c31990580571" path="/var/lib/kubelet/pods/7417b6c2-8fec-4720-95d5-c31990580571/volumes"
May 29 10:40:49.966760 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.966723    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ztkr5/must-gather-n5psh"]
May 29 10:40:49.967214 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967202    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="registry"
May 29 10:40:49.967283 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967220    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="registry"
May 29 10:40:49.967283 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967230    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="ui"
May 29 10:40:49.967283 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967239    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="ui"
May 29 10:40:49.967283 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967251    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="feast-apply"
May 29 10:40:49.967283 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967260    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="feast-apply"
May 29 10:40:49.967283 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967273    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="feast-apply"
May 29 10:40:49.967283 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967282    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="feast-apply"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967294    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="feast-init"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967303    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="feast-init"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967318    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="online"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967327    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="online"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967340    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="offline"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967348    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="offline"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967357    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="online"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967365    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="online"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967376    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="ui"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967384    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="ui"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967394    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="feast-init"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967402    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="feast-init"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967410    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="offline"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967418    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="offline"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967500    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="offline"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967512    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="online"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967524    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="online"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967533    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="registry"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967546    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="7417b6c2-8fec-4720-95d5-c31990580571" containerName="ui"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967555    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="offline"
May 29 10:40:49.967627 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.967566    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a333189-ccce-45de-83c9-8d489b883ff7" containerName="ui"
May 29 10:40:49.972209 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.972189    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:40:49.974395 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.974359    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ztkr5\"/\"openshift-service-ca.crt\""
May 29 10:40:49.975032 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.975006    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ztkr5\"/\"kube-root-ca.crt\""
May 29 10:40:49.975227 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.975205    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ztkr5\"/\"default-dockercfg-xlntw\""
May 29 10:40:49.976833 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:49.976811    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ztkr5/must-gather-n5psh"]
May 29 10:40:50.049054 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:50.049020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w9zw\" (UniqueName: \"kubernetes.io/projected/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-kube-api-access-4w9zw\") pod \"must-gather-n5psh\" (UID: \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\") " pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:40:50.049220 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:50.049087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-must-gather-output\") pod \"must-gather-n5psh\" (UID: \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\") " pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:40:50.150396 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:50.150358    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4w9zw\" (UniqueName: \"kubernetes.io/projected/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-kube-api-access-4w9zw\") pod \"must-gather-n5psh\" (UID: \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\") " pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:40:50.150577 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:50.150423    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-must-gather-output\") pod \"must-gather-n5psh\" (UID: \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\") " pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:40:50.150794 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:50.150774    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-must-gather-output\") pod \"must-gather-n5psh\" (UID: \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\") " pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:40:50.157561 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:50.157539    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w9zw\" (UniqueName: \"kubernetes.io/projected/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-kube-api-access-4w9zw\") pod \"must-gather-n5psh\" (UID: \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\") " pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:40:50.294424 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:50.294343    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:40:50.416980 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:50.416955    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ztkr5/must-gather-n5psh"]
May 29 10:40:50.420061 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:40:50.420029    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5077bd96_0b3c_48d6_bd61_754ec00ea7b5.slice/crio-363378cdc5764cec37a33fee48b3a85536d86a360cbcc4eb0d9093002142c2e9 WatchSource:0}: Error finding container 363378cdc5764cec37a33fee48b3a85536d86a360cbcc4eb0d9093002142c2e9: Status 404 returned error can't find the container with id 363378cdc5764cec37a33fee48b3a85536d86a360cbcc4eb0d9093002142c2e9
May 29 10:40:51.336513 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:51.336474    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztkr5/must-gather-n5psh" event={"ID":"5077bd96-0b3c-48d6-bd61-754ec00ea7b5","Type":"ContainerStarted","Data":"363378cdc5764cec37a33fee48b3a85536d86a360cbcc4eb0d9093002142c2e9"}
May 29 10:40:57.358838 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:57.358784    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztkr5/must-gather-n5psh" event={"ID":"5077bd96-0b3c-48d6-bd61-754ec00ea7b5","Type":"ContainerStarted","Data":"0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271"}
May 29 10:40:57.359285 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:57.358851    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztkr5/must-gather-n5psh" event={"ID":"5077bd96-0b3c-48d6-bd61-754ec00ea7b5","Type":"ContainerStarted","Data":"83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07"}
May 29 10:40:57.373732 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:40:57.373677    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-ztkr5/must-gather-n5psh" podStartSLOduration=2.34091261 podStartE2EDuration="8.373660086s" podCreationTimestamp="2026-05-29 10:40:49 +0000 UTC" firstStartedPulling="2026-05-29 10:40:50.421827731 +0000 UTC m=+1673.318387329" lastFinishedPulling="2026-05-29 10:40:56.454575205 +0000 UTC m=+1679.351134805" observedRunningTime="2026-05-29 10:40:57.371887505 +0000 UTC m=+1680.268447126" watchObservedRunningTime="2026-05-29 10:40:57.373660086 +0000 UTC m=+1680.270219706"
May 29 10:41:06.386532 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:06.386441    2562 generic.go:358] "Generic (PLEG): container finished" podID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" containerID="83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07" exitCode=0
May 29 10:41:06.386532 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:06.386518    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztkr5/must-gather-n5psh" event={"ID":"5077bd96-0b3c-48d6-bd61-754ec00ea7b5","Type":"ContainerDied","Data":"83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07"}
May 29 10:41:06.387001 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:06.386843    2562 scope.go:117] "RemoveContainer" containerID="83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07"
May 29 10:41:07.155658 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:07.155620    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ztkr5_must-gather-n5psh_5077bd96-0b3c-48d6-bd61-754ec00ea7b5/gather/0.log"
May 29 10:41:10.253212 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:10.253168    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x52b4_d7e789c0-dfb3-48d7-bd89-cbc8e59242c0/global-pull-secret-syncer/0.log"
May 29 10:41:10.307274 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:10.307235    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5j64w_17381d52-b51e-4d06-86d0-d814cdef85c5/konnectivity-agent/0.log"
May 29 10:41:10.408004 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:10.407973    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-181.ec2.internal_523418bbe89940c3ca4a0795cf595797/haproxy/0.log"
May 29 10:41:12.496384 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.496300    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-ztkr5/must-gather-n5psh"]
May 29 10:41:12.496803 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.496504    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-ztkr5/must-gather-n5psh" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" containerName="copy" containerID="cri-o://0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271" gracePeriod=2
May 29 10:41:12.498521 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.498485    2562 status_manager.go:895] "Failed to get status for pod" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" pod="openshift-must-gather-ztkr5/must-gather-n5psh" err="pods \"must-gather-n5psh\" is forbidden: User \"system:node:ip-10-0-132-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ztkr5\": no relationship found between node 'ip-10-0-132-181.ec2.internal' and this object"
May 29 10:41:12.499950 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.499924    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-ztkr5/must-gather-n5psh"]
May 29 10:41:12.728696 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.728670    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ztkr5_must-gather-n5psh_5077bd96-0b3c-48d6-bd61-754ec00ea7b5/copy/0.log"
May 29 10:41:12.728979 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.728965    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:41:12.730650 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.730598    2562 status_manager.go:895] "Failed to get status for pod" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" pod="openshift-must-gather-ztkr5/must-gather-n5psh" err="pods \"must-gather-n5psh\" is forbidden: User \"system:node:ip-10-0-132-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ztkr5\": no relationship found between node 'ip-10-0-132-181.ec2.internal' and this object"
May 29 10:41:12.860229 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.860201    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-must-gather-output\") pod \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\" (UID: \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\") "
May 29 10:41:12.860371 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.860265    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w9zw\" (UniqueName: \"kubernetes.io/projected/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-kube-api-access-4w9zw\") pod \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\" (UID: \"5077bd96-0b3c-48d6-bd61-754ec00ea7b5\") "
May 29 10:41:12.860596 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.860575    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "5077bd96-0b3c-48d6-bd61-754ec00ea7b5" (UID: "5077bd96-0b3c-48d6-bd61-754ec00ea7b5"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:41:12.862475 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.862454    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-kube-api-access-4w9zw" (OuterVolumeSpecName: "kube-api-access-4w9zw") pod "5077bd96-0b3c-48d6-bd61-754ec00ea7b5" (UID: "5077bd96-0b3c-48d6-bd61-754ec00ea7b5"). InnerVolumeSpecName "kube-api-access-4w9zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:41:12.961251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.961212    2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-must-gather-output\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:41:12.961251 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:12.961243    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w9zw\" (UniqueName: \"kubernetes.io/projected/5077bd96-0b3c-48d6-bd61-754ec00ea7b5-kube-api-access-4w9zw\") on node \"ip-10-0-132-181.ec2.internal\" DevicePath \"\""
May 29 10:41:13.357143 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.357111    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9bec710-e704-4f60-9a51-4c0f5f42fd2d/alertmanager/0.log"
May 29 10:41:13.384342 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.384318    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9bec710-e704-4f60-9a51-4c0f5f42fd2d/config-reloader/0.log"
May 29 10:41:13.408289 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.408265    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ztkr5_must-gather-n5psh_5077bd96-0b3c-48d6-bd61-754ec00ea7b5/copy/0.log"
May 29 10:41:13.408624 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.408599    2562 generic.go:358] "Generic (PLEG): container finished" podID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" containerID="0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271" exitCode=143
May 29 10:41:13.408761 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.408672    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztkr5/must-gather-n5psh"
May 29 10:41:13.408761 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.408693    2562 scope.go:117] "RemoveContainer" containerID="0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271"
May 29 10:41:13.410286 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.410262    2562 status_manager.go:895] "Failed to get status for pod" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" pod="openshift-must-gather-ztkr5/must-gather-n5psh" err="pods \"must-gather-n5psh\" is forbidden: User \"system:node:ip-10-0-132-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ztkr5\": no relationship found between node 'ip-10-0-132-181.ec2.internal' and this object"
May 29 10:41:13.415387 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.415364    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9bec710-e704-4f60-9a51-4c0f5f42fd2d/kube-rbac-proxy-web/0.log"
May 29 10:41:13.417311 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.417289    2562 scope.go:117] "RemoveContainer" containerID="83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07"
May 29 10:41:13.418803 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.418779    2562 status_manager.go:895] "Failed to get status for pod" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" pod="openshift-must-gather-ztkr5/must-gather-n5psh" err="pods \"must-gather-n5psh\" is forbidden: User \"system:node:ip-10-0-132-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ztkr5\": no relationship found between node 'ip-10-0-132-181.ec2.internal' and this object"
May 29 10:41:13.428462 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.428445    2562 scope.go:117] "RemoveContainer" containerID="0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271"
May 29 10:41:13.428752 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:41:13.428733    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271\": container with ID starting with 0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271 not found: ID does not exist" containerID="0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271"
May 29 10:41:13.428797 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.428761    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271"} err="failed to get container status \"0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271\": rpc error: code = NotFound desc = could not find container \"0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271\": container with ID starting with 0eeee1f2bc3932c17f35877687932e17fed45a5cf1a4f5f8e79e35a18bc8c271 not found: ID does not exist"
May 29 10:41:13.428797 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.428780    2562 scope.go:117] "RemoveContainer" containerID="83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07"
May 29 10:41:13.429026 ip-10-0-132-181 kubenswrapper[2562]: E0529 10:41:13.429004    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07\": container with ID starting with 83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07 not found: ID does not exist" containerID="83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07"
May 29 10:41:13.429074 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.429031    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07"} err="failed to get container status \"83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07\": rpc error: code = NotFound desc = could not find container \"83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07\": container with ID starting with 83262f4a88127b7a95b92f95b4e256e186f7aa98caf00866aedeb5277b933c07 not found: ID does not exist"
May 29 10:41:13.438217 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.438198    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9bec710-e704-4f60-9a51-4c0f5f42fd2d/kube-rbac-proxy/0.log"
May 29 10:41:13.462408 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.462386    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9bec710-e704-4f60-9a51-4c0f5f42fd2d/kube-rbac-proxy-metric/0.log"
May 29 10:41:13.488060 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.488031    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9bec710-e704-4f60-9a51-4c0f5f42fd2d/prom-label-proxy/0.log"
May 29 10:41:13.513806 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.513774    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9bec710-e704-4f60-9a51-4c0f5f42fd2d/init-config-reloader/0.log"
May 29 10:41:13.576653 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.576606    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-sbx5p_178970c5-e1e7-4175-9613-37ef79e22aab/kube-state-metrics/0.log"
May 29 10:41:13.597262 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.597237    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-sbx5p_178970c5-e1e7-4175-9613-37ef79e22aab/kube-rbac-proxy-main/0.log"
May 29 10:41:13.620974 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.620907    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-sbx5p_178970c5-e1e7-4175-9613-37ef79e22aab/kube-rbac-proxy-self/0.log"
May 29 10:41:13.647460 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.647429    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-797bb99f5d-sf4b2_db23805f-285b-4e55-8a4b-ab229fe6e126/metrics-server/0.log"
May 29 10:41:13.672202 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.672177    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-57r4k_8c45c144-f74e-441f-877d-ee7a7ebafaec/monitoring-plugin/0.log"
May 29 10:41:13.703617 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.703593    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" path="/var/lib/kubelet/pods/5077bd96-0b3c-48d6-bd61-754ec00ea7b5/volumes"
May 29 10:41:13.774369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.774344    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ddr92_5e768df9-bc5a-4816-9c9e-c5df5a4d8c35/node-exporter/0.log"
May 29 10:41:13.796151 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.796128    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ddr92_5e768df9-bc5a-4816-9c9e-c5df5a4d8c35/kube-rbac-proxy/0.log"
May 29 10:41:13.822245 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.822217    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ddr92_5e768df9-bc5a-4816-9c9e-c5df5a4d8c35/init-textfile/0.log"
May 29 10:41:13.920592 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.920518    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-qs7t7_78b1c61d-cdb6-4f3d-928d-8fc041990dad/kube-rbac-proxy-main/0.log"
May 29 10:41:13.943967 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.943928    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-qs7t7_78b1c61d-cdb6-4f3d-928d-8fc041990dad/kube-rbac-proxy-self/0.log"
May 29 10:41:13.965929 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.965902    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-qs7t7_78b1c61d-cdb6-4f3d-928d-8fc041990dad/openshift-state-metrics/0.log"
May 29 10:41:13.998913 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:13.998886    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f59f6b1-1862-42c8-b20c-aecb86a65037/prometheus/0.log"
May 29 10:41:14.020300 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.020277    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f59f6b1-1862-42c8-b20c-aecb86a65037/config-reloader/0.log"
May 29 10:41:14.045133 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.045105    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f59f6b1-1862-42c8-b20c-aecb86a65037/thanos-sidecar/0.log"
May 29 10:41:14.067992 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.067969    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f59f6b1-1862-42c8-b20c-aecb86a65037/kube-rbac-proxy-web/0.log"
May 29 10:41:14.092594 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.092568    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f59f6b1-1862-42c8-b20c-aecb86a65037/kube-rbac-proxy/0.log"
May 29 10:41:14.121200 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.121180    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f59f6b1-1862-42c8-b20c-aecb86a65037/kube-rbac-proxy-thanos/0.log"
May 29 10:41:14.147006 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.146983    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f59f6b1-1862-42c8-b20c-aecb86a65037/init-config-reloader/0.log"
May 29 10:41:14.176132 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.176050    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-d4dwc_95552ec2-9eee-4ee6-beee-1464cd7878ee/prometheus-operator/0.log"
May 29 10:41:14.195775 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.195750    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-d4dwc_95552ec2-9eee-4ee6-beee-1464cd7878ee/kube-rbac-proxy/0.log"
May 29 10:41:14.220968 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.220938    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-gdgdv_dabfe1b7-f676-4956-80e8-fb95d00ea4a9/prometheus-operator-admission-webhook/0.log"
May 29 10:41:14.253043 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.253009    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dcb655b7b-jpcww_a64ba713-1a70-4565-bd5c-eec6ce2774f9/telemeter-client/0.log"
May 29 10:41:14.276788 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.276765    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dcb655b7b-jpcww_a64ba713-1a70-4565-bd5c-eec6ce2774f9/reload/0.log"
May 29 10:41:14.299817 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.299784    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dcb655b7b-jpcww_a64ba713-1a70-4565-bd5c-eec6ce2774f9/kube-rbac-proxy/0.log"
May 29 10:41:14.334367 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.334340    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7d97cd7c-6tffh_ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6/thanos-query/0.log"
May 29 10:41:14.357890 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.357865    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7d97cd7c-6tffh_ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6/kube-rbac-proxy-web/0.log"
May 29 10:41:14.390329 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.390310    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7d97cd7c-6tffh_ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6/kube-rbac-proxy/0.log"
May 29 10:41:14.420253 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.420229    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7d97cd7c-6tffh_ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6/prom-label-proxy/0.log"
May 29 10:41:14.442815 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.442747    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7d97cd7c-6tffh_ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6/kube-rbac-proxy-rules/0.log"
May 29 10:41:14.466814 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:14.466788    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7d97cd7c-6tffh_ae8f7f20-2ba8-497b-9f40-3b9c28b37ac6/kube-rbac-proxy-metrics/0.log"
May 29 10:41:16.880228 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.880195    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"]
May 29 10:41:16.880577 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.880521    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" containerName="gather"
May 29 10:41:16.880577 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.880532    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" containerName="gather"
May 29 10:41:16.880577 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.880550    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" containerName="copy"
May 29 10:41:16.880577 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.880556    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" containerName="copy"
May 29 10:41:16.880737 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.880600    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" containerName="gather"
May 29 10:41:16.880737 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.880611    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="5077bd96-0b3c-48d6-bd61-754ec00ea7b5" containerName="copy"
May 29 10:41:16.883703 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.883687    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.885787 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.885766    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4ph8\"/\"kube-root-ca.crt\""
May 29 10:41:16.885898 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.885766    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4ph8\"/\"openshift-service-ca.crt\""
May 29 10:41:16.885898 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.885850    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t4ph8\"/\"default-dockercfg-kfcn2\""
May 29 10:41:16.891951 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.891927    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"]
May 29 10:41:16.892333 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.892314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvqw\" (UniqueName: \"kubernetes.io/projected/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-kube-api-access-fmvqw\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.892387 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.892350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-sys\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.892431 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.892400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-podres\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.892479 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.892465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-lib-modules\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.892520 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.892505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-proc\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.993768 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.993732    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-proc\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.993961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.993787    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvqw\" (UniqueName: \"kubernetes.io/projected/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-kube-api-access-fmvqw\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.993961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.993807    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-sys\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.993961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.993824    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-podres\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.993961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.993870    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-lib-modules\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.993961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.993879    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-proc\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.993961 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.993885    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-sys\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.994167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.993959    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-podres\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:16.994167 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:16.993972    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-lib-modules\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:17.001004 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.000977    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvqw\" (UniqueName: \"kubernetes.io/projected/99ca8f72-9956-4c46-ab38-b67fbf0b2d81-kube-api-access-fmvqw\") pod \"perf-node-gather-daemonset-vtss6\" (UID: \"99ca8f72-9956-4c46-ab38-b67fbf0b2d81\") " pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:17.194986 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.194901    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:17.330388 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.330327    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"]
May 29 10:41:17.333145 ip-10-0-132-181 kubenswrapper[2562]: W0529 10:41:17.333113    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod99ca8f72_9956_4c46_ab38_b67fbf0b2d81.slice/crio-a7918a8584bd95c7fd85cfca89715773c9a8992424cc785fce4f464d194b095b WatchSource:0}: Error finding container a7918a8584bd95c7fd85cfca89715773c9a8992424cc785fce4f464d194b095b: Status 404 returned error can't find the container with id a7918a8584bd95c7fd85cfca89715773c9a8992424cc785fce4f464d194b095b
May 29 10:41:17.334586 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.334568    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 10:41:17.408878 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.408855    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qxnsw_facacd1b-46f4-46af-9447-11f7a0245ab2/dns/0.log"
May 29 10:41:17.422205 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.422165    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6" event={"ID":"99ca8f72-9956-4c46-ab38-b67fbf0b2d81","Type":"ContainerStarted","Data":"f9c440c9a9e6bfc3bfba6351327896433e7895ca9f2edffb528f1019b2d53b1c"}
May 29 10:41:17.422336 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.422214    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6" event={"ID":"99ca8f72-9956-4c46-ab38-b67fbf0b2d81","Type":"ContainerStarted","Data":"a7918a8584bd95c7fd85cfca89715773c9a8992424cc785fce4f464d194b095b"}
May 29 10:41:17.422336 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.422246    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:17.430287 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.430256    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qxnsw_facacd1b-46f4-46af-9447-11f7a0245ab2/kube-rbac-proxy/0.log"
May 29 10:41:17.437310 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.437268    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6" podStartSLOduration=1.437253367 podStartE2EDuration="1.437253367s" podCreationTimestamp="2026-05-29 10:41:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:41:17.435228074 +0000 UTC m=+1700.331787693" watchObservedRunningTime="2026-05-29 10:41:17.437253367 +0000 UTC m=+1700.333812986"
May 29 10:41:17.518566 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.518456    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nd22r_27b9a97c-fe5a-4e4c-9979-e69acd4e390b/dns-node-resolver/0.log"
May 29 10:41:17.958191 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:17.958147    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxhx6_e131e33e-9e69-494c-ad8e-865e8f73e4b9/node-ca/0.log"
May 29 10:41:19.006956 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:19.006915    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8272x_19b92d9f-b8be-47e1-b184-0f0769490fd6/serve-healthcheck-canary/0.log"
May 29 10:41:19.516279 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:19.516246    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z69ss_88ed25c8-f538-4c47-82a7-f41c3ec07cf4/kube-rbac-proxy/0.log"
May 29 10:41:19.537053 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:19.537029    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z69ss_88ed25c8-f538-4c47-82a7-f41c3ec07cf4/exporter/0.log"
May 29 10:41:19.559236 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:19.559195    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z69ss_88ed25c8-f538-4c47-82a7-f41c3ec07cf4/extractor/0.log"
May 29 10:41:23.435390 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:23.435362    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t4ph8/perf-node-gather-daemonset-vtss6"
May 29 10:41:25.032997 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.032965    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5sf6_7f12b6be-7010-4361-a5e9-e161cb679061/kube-multus-additional-cni-plugins/0.log"
May 29 10:41:25.055905 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.055881    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5sf6_7f12b6be-7010-4361-a5e9-e161cb679061/egress-router-binary-copy/0.log"
May 29 10:41:25.076605 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.076583    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5sf6_7f12b6be-7010-4361-a5e9-e161cb679061/cni-plugins/0.log"
May 29 10:41:25.100001 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.099983    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5sf6_7f12b6be-7010-4361-a5e9-e161cb679061/bond-cni-plugin/0.log"
May 29 10:41:25.122199 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.122087    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5sf6_7f12b6be-7010-4361-a5e9-e161cb679061/routeoverride-cni/0.log"
May 29 10:41:25.143969 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.143951    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5sf6_7f12b6be-7010-4361-a5e9-e161cb679061/whereabouts-cni-bincopy/0.log"
May 29 10:41:25.168051 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.168026    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5sf6_7f12b6be-7010-4361-a5e9-e161cb679061/whereabouts-cni/0.log"
May 29 10:41:25.249369 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.249328    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v282r_27ff9c21-2605-4dd4-a989-e6161b491b8b/kube-multus/0.log"
May 29 10:41:25.363271 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.363244    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wk82g_ad608802-fd8d-49b6-945f-b56237be7996/network-metrics-daemon/0.log"
May 29 10:41:25.389157 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:25.389131    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wk82g_ad608802-fd8d-49b6-945f-b56237be7996/kube-rbac-proxy/0.log"
May 29 10:41:26.223741 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:26.223707    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-controller/0.log"
May 29 10:41:26.242780 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:26.242752    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/0.log"
May 29 10:41:26.257782 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:26.257760    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovn-acl-logging/1.log"
May 29 10:41:26.277015 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:26.276989    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/kube-rbac-proxy-node/0.log"
May 29 10:41:26.304169 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:26.304138    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/kube-rbac-proxy-ovn-metrics/0.log"
May 29 10:41:26.326093 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:26.326073    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/northd/0.log"
May 29 10:41:26.351769 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:26.351738    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/nbdb/0.log"
May 29 10:41:26.379285 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:26.379212    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/sbdb/0.log"
May 29 10:41:26.565953 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:26.565923    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kcfl_b63d125a-0397-4738-92bf-4798f84c67c6/ovnkube-controller/0.log"
May 29 10:41:28.079151 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:28.079121    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2j2z5_eb12e862-d565-46d8-b7da-df8d75835853/network-check-target-container/0.log"
May 29 10:41:28.966402 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:28.966371    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2dcgs_603d09a2-9f9f-49c1-b4d2-4d5174cb1f34/iptables-alerter/0.log"
May 29 10:41:29.826158 ip-10-0-132-181 kubenswrapper[2562]: I0529 10:41:29.826127    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-dzdll_ff455eab-532e-4f70-bb48-e56d679f294e/tuned/0.log"