May 29 10:47:49.105424 ip-10-0-142-12 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 29 10:47:49.105436 ip-10-0-142-12 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 29 10:47:49.105445 ip-10-0-142-12 systemd[1]: kubelet.service: Failed with result 'resources'.
May 29 10:47:49.105744 ip-10-0-142-12 systemd[1]: Failed to start Kubernetes Kubelet.
May 29 10:47:59.235597 ip-10-0-142-12 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 29 10:47:59.235621 ip-10-0-142-12 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 987a053ca16b48c292786961b7c24ee8 --
May 29 10:50:20.988631 ip-10-0-142-12 systemd[1]: Starting Kubernetes Kubelet...
May 29 10:50:21.494613 ip-10-0-142-12 kubenswrapper[2547]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 10:50:21.494613 ip-10-0-142-12 kubenswrapper[2547]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 29 10:50:21.494613 ip-10-0-142-12 kubenswrapper[2547]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 10:50:21.494613 ip-10-0-142-12 kubenswrapper[2547]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 29 10:50:21.494613 ip-10-0-142-12 kubenswrapper[2547]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 10:50:21.496510 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.496424    2547 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 29 10:50:21.500787 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500769    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:50:21.500787 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500787    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500791    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500795    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500798    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500801    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500804    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500807    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500810    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500813    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500816    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500819    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500823    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500826    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500829    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500833    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500837    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500840    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500844    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500847    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500850    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:50:21.500855 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500852    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500856    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500859    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500862    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500865    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500868    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500870    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500873    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500876    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500878    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500881    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500883    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500886    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500889    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500892    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500895    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500898    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500901    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500903    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500906    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:50:21.501442 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500908    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500912    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500914    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500917    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500919    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500922    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500924    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500927    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500929    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500932    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500934    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500937    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500939    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500942    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500946    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500949    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500952    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500954    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500957    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500959    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:50:21.501954 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500962    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500965    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500967    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500971    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500974    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500977    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500980    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500982    2547 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500985    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500988    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500990    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500993    2547 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.500997    2547 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501000    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501003    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501005    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501008    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501010    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501013    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:50:21.502460 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501016    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501018    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501021    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501023    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501026    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501028    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501454    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501462    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501466    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501470    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501473    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501476    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501479    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501482    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501485    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501487    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501490    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501493    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501496    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:50:21.502928 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501499    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501501    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501504    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501506    2547 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501509    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501514    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501517    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501520    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501523    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501525    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501528    2547 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501531    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501534    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501536    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501539    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501541    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501543    2547 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501546    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501549    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501551    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:50:21.503452 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501554    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501556    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501559    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501562    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501564    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501567    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501569    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501572    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501574    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501577    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501579    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501582    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501584    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501587    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501589    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501592    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501594    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501597    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501602    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501605    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:50:21.503949 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501608    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501610    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501614    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501617    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501619    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501621    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501624    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501626    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501630    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501632    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501634    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501637    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501640    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501642    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501645    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501648    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501650    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501654    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501658    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501661    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:50:21.504448 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501664    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501668    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501670    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501673    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501676    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501678    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501680    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501683    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501685    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501688    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501690    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501693    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.501696    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501774    2547 flags.go:64] FLAG: --address="0.0.0.0"
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501782    2547 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501789    2547 flags.go:64] FLAG: --anonymous-auth="true"
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501794    2547 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501800    2547 flags.go:64] FLAG: --authentication-token-webhook="false"
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501803    2547 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501810    2547 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501814    2547 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 29 10:50:21.504936 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501819    2547 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501822    2547 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501825    2547 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501829    2547 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501832    2547 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501835    2547 flags.go:64] FLAG: --cgroup-root=""
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501838    2547 flags.go:64] FLAG: --cgroups-per-qos="true"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501841    2547 flags.go:64] FLAG: --client-ca-file=""
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501844    2547 flags.go:64] FLAG: --cloud-config=""
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501847    2547 flags.go:64] FLAG: --cloud-provider="external"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501850    2547 flags.go:64] FLAG: --cluster-dns="[]"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501854    2547 flags.go:64] FLAG: --cluster-domain=""
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501857    2547 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501860    2547 flags.go:64] FLAG: --config-dir=""
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501863    2547 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501867    2547 flags.go:64] FLAG: --container-log-max-files="5"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501871    2547 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501874    2547 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501877    2547 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501880    2547 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501883    2547 flags.go:64] FLAG: --contention-profiling="false"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501886    2547 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501889    2547 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501897    2547 flags.go:64] FLAG: --cpu-manager-policy="none"
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501900    2547 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 29 10:50:21.505472 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501904    2547 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501908    2547 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501911    2547 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501914    2547 flags.go:64] FLAG: --enable-load-reader="false"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501917    2547 flags.go:64] FLAG: --enable-server="true"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501920    2547 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501925    2547 flags.go:64] FLAG: --event-burst="100"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501929    2547 flags.go:64] FLAG: --event-qps="50"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501932    2547 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501935    2547 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501938    2547 flags.go:64] FLAG: --eviction-hard=""
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501942    2547 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501945    2547 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501948    2547 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501951    2547 flags.go:64] FLAG: --eviction-soft=""
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501954    2547 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501957    2547 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501960    2547 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501963    2547 flags.go:64] FLAG: --experimental-mounter-path=""
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501966    2547 flags.go:64] FLAG: --fail-cgroupv1="false"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501969    2547 flags.go:64] FLAG: --fail-swap-on="true"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501972    2547 flags.go:64] FLAG: --feature-gates=""
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501975    2547 flags.go:64] FLAG: --file-check-frequency="20s"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501978    2547 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501981    2547 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 29 10:50:21.506079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501985    2547 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501988    2547 flags.go:64] FLAG: --healthz-port="10248"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501991    2547 flags.go:64] FLAG: --help="false"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501994    2547 flags.go:64] FLAG: --hostname-override="ip-10-0-142-12.ec2.internal"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501997    2547 flags.go:64] FLAG: --housekeeping-interval="10s"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.501999    2547 flags.go:64] FLAG: --http-check-frequency="20s"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502006    2547 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502011    2547 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502014    2547 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502017    2547 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502020    2547 flags.go:64] FLAG: --image-service-endpoint=""
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502023    2547 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502026    2547 flags.go:64] FLAG: --kube-api-burst="100"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502029    2547 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502032    2547 flags.go:64] FLAG: --kube-api-qps="50"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502035    2547 flags.go:64] FLAG: --kube-reserved=""
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502038    2547 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502041    2547 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502044    2547 flags.go:64] FLAG: --kubelet-cgroups=""
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502047    2547 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502050    2547 flags.go:64] FLAG: --lock-file=""
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502053    2547 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502055    2547 flags.go:64] FLAG: --log-flush-frequency="5s"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502059    2547 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 29 10:50:21.506705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502064    2547 flags.go:64] FLAG: --log-json-split-stream="false"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502067    2547 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502070    2547 flags.go:64] FLAG: --log-text-split-stream="false"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502072    2547 flags.go:64] FLAG: --logging-format="text"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502075    2547 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502078    2547 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502081    2547 flags.go:64] FLAG: --manifest-url=""
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502084    2547 flags.go:64] FLAG: --manifest-url-header=""
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502089    2547 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502092    2547 flags.go:64] FLAG: --max-open-files="1000000"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502096    2547 flags.go:64] FLAG: --max-pods="110"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502099    2547 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502102    2547 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502105    2547 flags.go:64] FLAG: --memory-manager-policy="None"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502108    2547 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502111    2547 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502114    2547 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502117    2547 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502126    2547 flags.go:64] FLAG: --node-status-max-images="50"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502129    2547 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502132    2547 flags.go:64] FLAG: --oom-score-adj="-999"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502135    2547 flags.go:64] FLAG: --pod-cidr=""
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502138    2547 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 29 10:50:21.507308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502144    2547 flags.go:64] FLAG: --pod-manifest-path=""
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502147    2547 flags.go:64] FLAG: --pod-max-pids="-1"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502151    2547 flags.go:64] FLAG: --pods-per-core="0"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502154    2547 flags.go:64] FLAG: --port="10250"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502157    2547 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502160    2547 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040bf437300111b8e"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502163    2547 flags.go:64] FLAG: --qos-reserved=""
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502166    2547 flags.go:64] FLAG: --read-only-port="10255"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502169    2547 flags.go:64] FLAG: --register-node="true"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502172    2547 flags.go:64] FLAG: --register-schedulable="true"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502175    2547 flags.go:64] FLAG: --register-with-taints=""
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502179    2547 flags.go:64] FLAG: --registry-burst="10"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502181    2547 flags.go:64] FLAG: --registry-qps="5"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502184    2547 flags.go:64] FLAG: --reserved-cpus=""
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502187    2547 flags.go:64] FLAG: --reserved-memory=""
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502191    2547 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502194    2547 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502200    2547 flags.go:64] FLAG: --rotate-certificates="false"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502203    2547 flags.go:64] FLAG: --rotate-server-certificates="false"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502206    2547 flags.go:64] FLAG: --runonce="false"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502209    2547 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502212    2547 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502215    2547 flags.go:64] FLAG: --seccomp-default="false"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502218    2547 flags.go:64] FLAG: --serialize-image-pulls="true"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502221    2547 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502225    2547 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 29 10:50:21.507867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502229    2547 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502232    2547 flags.go:64] FLAG: --storage-driver-password="root"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502248    2547 flags.go:64] FLAG: --storage-driver-secure="false"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502251    2547 flags.go:64] FLAG: --storage-driver-table="stats"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502254    2547 flags.go:64] FLAG: --storage-driver-user="root"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502257    2547 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502263    2547 flags.go:64] FLAG: --sync-frequency="1m0s"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502266    2547 flags.go:64] FLAG: --system-cgroups=""
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502268    2547 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502275    2547 flags.go:64] FLAG: --system-reserved-cgroup=""
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502278    2547 flags.go:64] FLAG: --tls-cert-file=""
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502281    2547 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502285    2547 flags.go:64] FLAG: --tls-min-version=""
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502289    2547 flags.go:64] FLAG: --tls-private-key-file=""
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502292    2547 flags.go:64] FLAG: --topology-manager-policy="none"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502295    2547 flags.go:64] FLAG: --topology-manager-policy-options=""
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502298    2547 flags.go:64] FLAG: --topology-manager-scope="container"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502301    2547 flags.go:64] FLAG: --v="2"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502305    2547 flags.go:64] FLAG: --version="false"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502310    2547 flags.go:64] FLAG: --vmodule=""
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502314    2547 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502318    2547 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502428    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502434    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:50:21.508567 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502438    2547 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502441    2547 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502443    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502446    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502449    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502451    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502454    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502456    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502460    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502462    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502465    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502468    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502471    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502473    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502477    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502480    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502483    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502485    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502488    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502490    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:50:21.509164 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502493    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502495    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502498    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502501    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502503    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502506    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502508    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502511    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502514    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502516    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502519    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502522    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502525    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502528    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502531    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502533    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502536    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502538    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502541    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502543    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:50:21.509695 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502546    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502549    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502553    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502557    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502560    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502563    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502567    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502570    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502573    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502576    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502579    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502581    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502584    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502586    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502589    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502592    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502594    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502597    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502599    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:50:21.510191 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502602    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502604    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502607    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502609    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502617    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502619    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502622    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502624    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502627    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502630    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502632    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502635    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502637    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502640    2547 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502642    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502645    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502648    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502652    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502655    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502659    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:50:21.510684 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502662    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502665    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502667    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502670    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.502673    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.502678    2547 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.509830    2547 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.509848    2547 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509902    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509908    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509911    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509915    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509918    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509921    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509924    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509927    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:50:21.511187 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509930    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509932    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509935    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509938    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509942    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509947    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509951    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509956    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509959    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509962    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509966    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509970    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509973    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509975    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509978    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509981    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509984    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509987    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509989    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:50:21.511610 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509992    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509994    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.509998    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510000    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510011    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510014    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510017    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510019    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510022    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510025    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510027    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510030    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510032    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510035    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510037    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510039    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510042    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510044    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510047    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510050    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:50:21.512115 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510052    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510055    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510057    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510060    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510062    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510065    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510067    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510070    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510072    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510075    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510077    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510080    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510082    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510084    2547 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510087    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510089    2547 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510093    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510101    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510104    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510107    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:50:21.512726 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510109    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510112    2547 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510115    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510118    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510120    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510122    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510125    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510127    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510130    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510133    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510135    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510137    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510140    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510143    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510145    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510148    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510151    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510153    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:50:21.513224 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510156    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.510161    2547 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510313    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510319    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510322    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510325    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510328    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510331    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510334    2547 feature_gate.go:328] unrecognized feature gate: Example
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510336    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510339    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510343    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510350    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510353    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510356    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510359    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 10:50:21.513670 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510361    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510364    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510367    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510369    2547 feature_gate.go:328] unrecognized feature gate: Example2
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510372    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510374    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510377    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510380    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510382    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510385    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510387    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510390    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510392    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510395    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510398    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510403    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510406    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510409    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510412    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 10:50:21.514073 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510414    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510418    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510422    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510425    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510428    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510431    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510434    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510437    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510440    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510443    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510451    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510453    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510456    2547 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510459    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510461    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510464    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510466    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510469    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510471    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 10:50:21.514541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510474    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510476    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510479    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510481    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510484    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510486    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510489    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510491    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510494    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510496    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510499    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510501    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510504    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510506    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510509    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510511    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510513    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510516    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510519    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510521    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 10:50:21.515046 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510524    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510527    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510530    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510532    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510540    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510543    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510546    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510549    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510551    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510553    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510556    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510558    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510561    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:21.510564    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.510569    2547 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 10:50:21.515584 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.511495    2547 server.go:962] "Client rotation is on, will bootstrap in background"
May 29 10:50:21.516013 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.515746    2547 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 29 10:50:21.516875 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.516861    2547 server.go:1019] "Starting client certificate rotation"
May 29 10:50:21.517001 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.516984    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 29 10:50:21.517831 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.517812    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 29 10:50:21.546214 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.546182    2547 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 29 10:50:21.553556 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.553501    2547 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 29 10:50:21.565373 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.565349    2547 log.go:25] "Validated CRI v1 runtime API"
May 29 10:50:21.571856 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.571831    2547 log.go:25] "Validated CRI v1 image API"
May 29 10:50:21.573467 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.573439    2547 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 29 10:50:21.578766 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.578744    2547 fs.go:135] Filesystem UUIDs: map[4c2e518a-6d45-483d-9a12-7bfe0fe2b533:/dev/nvme0n1p3 75031033-f88f-4f5c-81b2-5185407331c7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
May 29 10:50:21.578855 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.578764    2547 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 29 10:50:21.580728 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.580708    2547 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 29 10:50:21.583392 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.583275    2547 manager.go:217] Machine: {Timestamp:2026-05-29 10:50:21.582736885 +0000 UTC m=+0.459299104 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099425 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2be89c5bfce055915a90b20dd49390 SystemUUID:ec2be89c-5bfc-e055-915a-90b20dd49390 BootID:987a053c-a16b-48c2-9278-6961b7c24ee8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:14:4b:d7:c0:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:14:4b:d7:c0:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:91:85:ac:6a:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 29 10:50:21.583392 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.583384    2547 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 29 10:50:21.583521 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.583462    2547 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 29 10:50:21.586638 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.586607    2547 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 29 10:50:21.586824 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.586640    2547 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-12.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 29 10:50:21.586874 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.586842    2547 topology_manager.go:138] "Creating topology manager with none policy"
May 29 10:50:21.586874 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.586853    2547 container_manager_linux.go:306] "Creating device plugin manager"
May 29 10:50:21.586874 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.586860    2547 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 29 10:50:21.587785 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.587775    2547 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 29 10:50:21.590456 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.590445    2547 state_mem.go:36] "Initialized new in-memory state store"
May 29 10:50:21.590574 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.590563    2547 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 29 10:50:21.593374 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.593364    2547 kubelet.go:491] "Attempting to sync node with API server"
May 29 10:50:21.593408 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.593382    2547 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 29 10:50:21.593408 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.593394    2547 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 29 10:50:21.593408 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.593402    2547 kubelet.go:397] "Adding apiserver pod source"
May 29 10:50:21.593542 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.593411    2547 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 29 10:50:21.595018 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.594999    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 29 10:50:21.595018 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.595019    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 29 10:50:21.596865 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.596845    2547 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fr2qw"
May 29 10:50:21.598565 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.598551    2547 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 29 10:50:21.600902 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.600888    2547 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 29 10:50:21.601851 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601838    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 29 10:50:21.601901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601854    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 29 10:50:21.601901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601860    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 29 10:50:21.601901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601865    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 29 10:50:21.601901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601874    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 29 10:50:21.601901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601880    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 29 10:50:21.601901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601886    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 29 10:50:21.601901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601897    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 29 10:50:21.601901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601904    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 29 10:50:21.602112 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601911    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 29 10:50:21.602112 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.601923    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 29 10:50:21.602371 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.602339    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 29 10:50:21.602455 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.602375    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 29 10:50:21.602455 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.602430    2547 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 29 10:50:21.605122 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.605082    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fr2qw"
May 29 10:50:21.607092 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.607067    2547 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-12.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 29 10:50:21.607281 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.607258    2547 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-12.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 29 10:50:21.607334 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.607312    2547 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 29 10:50:21.608742 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.608725    2547 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 29 10:50:21.608809 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.608778    2547 server.go:1295] "Started kubelet"
May 29 10:50:21.608943 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.608834    2547 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 29 10:50:21.608978 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.608915    2547 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 29 10:50:21.609005 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.608989    2547 server_v1.go:47] "podresources" method="list" useActivePods=true
May 29 10:50:21.609527 ip-10-0-142-12 systemd[1]: Started Kubernetes Kubelet.
May 29 10:50:21.611158 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.611119    2547 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 29 10:50:21.611405 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.611391    2547 server.go:317] "Adding debug handlers to kubelet server"
May 29 10:50:21.615075 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.613945    2547 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-12.ec2.internal.18b403b35cc6a871  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-12.ec2.internal,UID:ip-10-0-142-12.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-12.ec2.internal,},FirstTimestamp:2026-05-29 10:50:21.608740977 +0000 UTC m=+0.485303197,LastTimestamp:2026-05-29 10:50:21.608740977 +0000 UTC m=+0.485303197,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-12.ec2.internal,}"
May 29 10:50:21.617835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.617810    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 29 10:50:21.618351 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.618332    2547 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 29 10:50:21.618574 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.618561    2547 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 29 10:50:21.618646 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.618619    2547 volume_manager.go:295] "The desired_state_of_world populator starts"
May 29 10:50:21.618701 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.618649    2547 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 29 10:50:21.618787 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.618774    2547 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 29 10:50:21.618869 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.618858    2547 reconstruct.go:97] "Volume reconstruction finished"
May 29 10:50:21.618922 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.618870    2547 reconciler.go:26] "Reconciler: start to sync state"
May 29 10:50:21.619715 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.619682    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:21.620731 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.620713    2547 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 29 10:50:21.620827 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.620791    2547 factory.go:55] Registering systemd factory
May 29 10:50:21.620827 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.620805    2547 factory.go:223] Registration of the systemd container factory successfully
May 29 10:50:21.621134 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.621118    2547 factory.go:153] Registering CRI-O factory
May 29 10:50:21.621185 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.621137    2547 factory.go:223] Registration of the crio container factory successfully
May 29 10:50:21.621185 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.621155    2547 factory.go:103] Registering Raw factory
May 29 10:50:21.621185 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.621168    2547 manager.go:1196] Started watching for new ooms in manager
May 29 10:50:21.621806 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.621782    2547 manager.go:319] Starting recovery of all containers
May 29 10:50:21.630278 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.630107    2547 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:50:21.631700 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.631682    2547 manager.go:324] Recovery completed
May 29 10:50:21.633321 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.633300    2547 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-12.ec2.internal\" not found" node="ip-10-0-142-12.ec2.internal"
May 29 10:50:21.637018 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.637005    2547 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:50:21.638707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.638692    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:50:21.638785 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.638717    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:50:21.638785 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.638728    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID"
May 29 10:50:21.639284 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.639269    2547 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 29 10:50:21.639335 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.639283    2547 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 29 10:50:21.639335 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.639302    2547 state_mem.go:36] "Initialized new in-memory state store"
May 29 10:50:21.643107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.643095    2547 policy_none.go:49] "None policy: Start"
May 29 10:50:21.643144 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.643111    2547 memory_manager.go:186] "Starting memorymanager" policy="None"
May 29 10:50:21.643144 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.643122    2547 state_mem.go:35] "Initializing new in-memory state store"
May 29 10:50:21.670881 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.670861    2547 manager.go:341] "Starting Device Plugin manager"
May 29 10:50:21.670979 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.670893    2547 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 29 10:50:21.670979 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.670906    2547 server.go:85] "Starting device plugin registration server"
May 29 10:50:21.671152 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.671141    2547 eviction_manager.go:189] "Eviction manager: starting control loop"
May 29 10:50:21.671188 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.671154    2547 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 29 10:50:21.671269 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.671226    2547 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 29 10:50:21.671323 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.671317    2547 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 29 10:50:21.671323 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.671323    2547 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 29 10:50:21.671889 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.671872    2547 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 29 10:50:21.671994 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.671904    2547 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:21.710920 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.710881    2547 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 29 10:50:21.715085 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.712121    2547 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 29 10:50:21.715085 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.712144    2547 status_manager.go:230] "Starting to sync pod status with apiserver"
May 29 10:50:21.715085 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.712162    2547 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 29 10:50:21.715085 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.712168    2547 kubelet.go:2451] "Starting kubelet main sync loop"
May 29 10:50:21.715085 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.712206    2547 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 29 10:50:21.715085 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.714433    2547 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:50:21.771520 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.771439    2547 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:50:21.772729 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.772711    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:50:21.772831 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.772741    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:50:21.772831 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.772753    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID"
May 29 10:50:21.772831 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.772778    2547 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-12.ec2.internal"
May 29 10:50:21.778566 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.778551    2547 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-12.ec2.internal"
May 29 10:50:21.778612 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.778576    2547 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-12.ec2.internal\": node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:21.802976 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.802952    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:21.813038 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.813008    2547 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal"]
May 29 10:50:21.813111 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.813102    2547 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:50:21.813996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.813979    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:50:21.814099 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.814012    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:50:21.814099 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.814027    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID"
May 29 10:50:21.815362 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.815347    2547 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:50:21.815511 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.815496    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.815560 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.815538    2547 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:50:21.816073 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.816059    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:50:21.816073 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.816068    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:50:21.816189 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.816093    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:50:21.816189 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.816103    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID"
May 29 10:50:21.816189 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.816114    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:50:21.816189 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.816146    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID"
May 29 10:50:21.817225 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.817208    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.817321 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.817260    2547 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 10:50:21.817875 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.817860    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory"
May 29 10:50:21.817941 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.817886    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure"
May 29 10:50:21.817941 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.817898    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID"
May 29 10:50:21.819295 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.819275    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c7eed8448233ff397c88b5c98ff0229-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"6c7eed8448233ff397c88b5c98ff0229\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.819366 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.819310    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f693add312221a196297edaec6e814ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-12.ec2.internal\" (UID: \"f693add312221a196297edaec6e814ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.819366 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.819341    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c7eed8448233ff397c88b5c98ff0229-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"6c7eed8448233ff397c88b5c98ff0229\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.844059 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.844033    2547 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-12.ec2.internal\" not found" node="ip-10-0-142-12.ec2.internal"
May 29 10:50:21.848556 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.848535    2547 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-12.ec2.internal\" not found" node="ip-10-0-142-12.ec2.internal"
May 29 10:50:21.903272 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:21.903230    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:21.919641 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.919614    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c7eed8448233ff397c88b5c98ff0229-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"6c7eed8448233ff397c88b5c98ff0229\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.919766 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.919652    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c7eed8448233ff397c88b5c98ff0229-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"6c7eed8448233ff397c88b5c98ff0229\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.919766 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.919678    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f693add312221a196297edaec6e814ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-12.ec2.internal\" (UID: \"f693add312221a196297edaec6e814ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.919766 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.919679    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c7eed8448233ff397c88b5c98ff0229-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"6c7eed8448233ff397c88b5c98ff0229\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.919766 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.919697    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c7eed8448233ff397c88b5c98ff0229-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"6c7eed8448233ff397c88b5c98ff0229\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal"
May 29 10:50:21.919766 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:21.919703    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f693add312221a196297edaec6e814ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-12.ec2.internal\" (UID: \"f693add312221a196297edaec6e814ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal"
May 29 10:50:22.003945 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.003897    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.104671 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.104594    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.147012 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.146989    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal"
May 29 10:50:22.151826 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.151807    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal"
May 29 10:50:22.205562 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.205525    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.306081 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.306042    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.406564 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.406493    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.507176 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.507144    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.516304 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.516280    2547 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 29 10:50:22.516515 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.516425    2547 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 29 10:50:22.516515 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.516474    2547 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 29 10:50:22.607249 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.607193    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-28 10:45:21 +0000 UTC" deadline="2027-11-13 20:19:24.93987108 +0000 UTC"
May 29 10:50:22.607249 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.607234    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12801h29m2.332640354s"
May 29 10:50:22.607249 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.607206    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.617947 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.617922    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 29 10:50:22.628486 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.628464    2547 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 29 10:50:22.649648 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.649626    2547 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-47494"
May 29 10:50:22.657325 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.657266    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-47494"
May 29 10:50:22.658523 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:22.658499    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7eed8448233ff397c88b5c98ff0229.slice/crio-762f1ebca2a2fb598aed6846b9010915389fa47d24bfb9f9d431693e8baff20b WatchSource:0}: Error finding container 762f1ebca2a2fb598aed6846b9010915389fa47d24bfb9f9d431693e8baff20b: Status 404 returned error can't find the container with id 762f1ebca2a2fb598aed6846b9010915389fa47d24bfb9f9d431693e8baff20b
May 29 10:50:22.658750 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:22.658724    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf693add312221a196297edaec6e814ca.slice/crio-2afd93d8088b3ea54e55934fbe9637a645a54f0a798143259f3e9af0d2d26353 WatchSource:0}: Error finding container 2afd93d8088b3ea54e55934fbe9637a645a54f0a798143259f3e9af0d2d26353: Status 404 returned error can't find the container with id 2afd93d8088b3ea54e55934fbe9637a645a54f0a798143259f3e9af0d2d26353
May 29 10:50:22.663087 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.663069    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 10:50:22.708150 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.708107    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.715083 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.715031    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" event={"ID":"6c7eed8448233ff397c88b5c98ff0229","Type":"ContainerStarted","Data":"762f1ebca2a2fb598aed6846b9010915389fa47d24bfb9f9d431693e8baff20b"}
May 29 10:50:22.715882 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.715863    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" event={"ID":"f693add312221a196297edaec6e814ca","Type":"ContainerStarted","Data":"2afd93d8088b3ea54e55934fbe9637a645a54f0a798143259f3e9af0d2d26353"}
May 29 10:50:22.808402 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.808364    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.908935 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:22.908852    2547 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found"
May 29 10:50:22.925791 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:22.924867    2547 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:50:23.004479 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.004452    2547 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:50:23.019120 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.019096    2547 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal"
May 29 10:50:23.034216 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.034195    2547 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 29 10:50:23.035395 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.035381    2547 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal"
May 29 10:50:23.041686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.041662    2547 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 29 10:50:23.551626 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.551597    2547 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:50:23.594348 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.594310    2547 apiserver.go:52] "Watching apiserver"
May 29 10:50:23.601594 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.601566    2547 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 29 10:50:23.601997 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.601971    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-zlmdr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6","openshift-cluster-node-tuning-operator/tuned-x84w9","openshift-image-registry/node-ca-2n89j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-c8682","kube-system/konnectivity-agent-nxddb","kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal","openshift-dns/node-resolver-ghsp2","openshift-multus/multus-97zrn","openshift-multus/multus-additional-cni-plugins-qm79t","openshift-multus/network-metrics-daemon-fqncf","openshift-network-diagnostics/network-check-target-4524p"]
May 29 10:50:23.604612 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.604594    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:23.606766 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.606638    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 29 10:50:23.606766 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.606658    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n9ggm\""
May 29 10:50:23.606766 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.606664    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 29 10:50:23.608786 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.608748    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.610852 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.610749    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 29 10:50:23.610933 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.610861    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 29 10:50:23.610933 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.610880    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5z24c\""
May 29 10:50:23.611034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.610935    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.611081 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.611066    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 29 10:50:23.613191 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.613053    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 29 10:50:23.613279 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.613218    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 29 10:50:23.613279 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.613221    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.613382 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.613356    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tptrn\""
May 29 10:50:23.615330 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.615308    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 29 10:50:23.615440 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.615403    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 29 10:50:23.615497 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.615482    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl22f\""
May 29 10:50:23.615497 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.615484    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 29 10:50:23.615607 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.615519    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.617671 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.617649    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.617768 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.617749    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.619530 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.619504    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 29 10:50:23.620435 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.620179    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9h9sc\""
May 29 10:50:23.620435 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.620284    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 29 10:50:23.620543 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.620509    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 29 10:50:23.621770 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.620734    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 29 10:50:23.621770 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.620882    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 29 10:50:23.621770 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.620887    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 29 10:50:23.621770 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.621703    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 29 10:50:23.623259 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.622465    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 29 10:50:23.623259 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.622530    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 29 10:50:23.623259 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.622604    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pszjn\""
May 29 10:50:23.623259 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.623080    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 29 10:50:23.624071 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.624053    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkwkw\""
May 29 10:50:23.624394 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.624372    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-97zrn"
May 29 10:50:23.624566 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.624544    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.625495 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.625477    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 29 10:50:23.626917 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.626897    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 29 10:50:23.627151 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.627134    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 29 10:50:23.627582 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.627562    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 29 10:50:23.627713 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.627694    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dxhdp\""
May 29 10:50:23.627795 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.627719    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 29 10:50:23.627795 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.627775    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 29 10:50:23.627899 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.627797    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 29 10:50:23.627899 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.627893    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m2tq9\""
May 29 10:50:23.628193 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.628165    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:23.628301 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:23.628277    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:23.630711 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630692    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8e00f00-647c-463b-b317-aa670bde2fd4-host\") pod \"node-ca-2n89j\" (UID: \"e8e00f00-647c-463b-b317-aa670bde2fd4\") " pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.630808 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630712    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:23.630808 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630718    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-cnibin\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.630808 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630734    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-sysctl-d\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.630808 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630755    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-run-openvswitch\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.630808 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:23.630768    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:23.630808 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630785    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lcb9\" (UniqueName: \"kubernetes.io/projected/e8e00f00-647c-463b-b317-aa670bde2fd4-kube-api-access-2lcb9\") pod \"node-ca-2n89j\" (UID: \"e8e00f00-647c-463b-b317-aa670bde2fd4\") " pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.631107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630837    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-daemon-config\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.631107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630891    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-systemd\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.631107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630919    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-lib-modules\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.631107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630947    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a152d23c-60e9-432e-98c3-799273b117fd-ovn-node-metrics-cert\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.631107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.630983    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-system-cni-dir\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.631107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631009    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-cni-dir\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.631107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631034    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-modprobe-d\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.631107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631074    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6-agent-certs\") pod \"konnectivity-agent-nxddb\" (UID: \"a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6\") " pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:23.631107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631098    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6-konnectivity-ca\") pod \"konnectivity-agent-nxddb\" (UID: \"a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6\") " pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631121    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-run-systemd\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631145    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-cni-netd\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631195    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a152d23c-60e9-432e-98c3-799273b117fd-ovnkube-config\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631254    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-registration-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631283    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-sys-fs\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631305    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d00bb862-d044-4f60-8f8e-75f3e539d7a9-host-slash\") pod \"iptables-alerter-zlmdr\" (UID: \"d00bb862-d044-4f60-8f8e-75f3e539d7a9\") " pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631330    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041-tmp-dir\") pod \"node-resolver-ghsp2\" (UID: \"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041\") " pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631356    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-var-lib-cni-bin\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631384    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-socket-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631410    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-sysctl-conf\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631434    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-run\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631459    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-log-socket\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631482    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a152d23c-60e9-432e-98c3-799273b117fd-ovnkube-script-lib\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.631540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631505    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-socket-dir-parent\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631551    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a152d23c-60e9-432e-98c3-799273b117fd-env-overrides\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631576    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-run-k8s-cni-cncf-io\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631613    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-etc-kubernetes\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631638    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-slash\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631669    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-cni-bin\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631708    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-var-lib-cni-multus\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631750    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44zr5\" (UniqueName: \"kubernetes.io/projected/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-kube-api-access-44zr5\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631778    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-kubelet\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631805    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67r68\" (UniqueName: \"kubernetes.io/projected/a152d23c-60e9-432e-98c3-799273b117fd-kube-api-access-67r68\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631829    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttnn\" (UniqueName: \"kubernetes.io/projected/ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041-kube-api-access-zttnn\") pod \"node-resolver-ghsp2\" (UID: \"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041\") " pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631853    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-run-netns\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631889    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-etc-openvswitch\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631919    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631944    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d00bb862-d044-4f60-8f8e-75f3e539d7a9-iptables-alerter-script\") pod \"iptables-alerter-zlmdr\" (UID: \"d00bb862-d044-4f60-8f8e-75f3e539d7a9\") " pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631969    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-var-lib-kubelet\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.631995    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-run-ovn\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632020    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-os-release\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632045    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-conf-dir\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632082    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-var-lib-kubelet\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632113    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-node-log\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632138    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-run-netns\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632161    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-hostroot\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632184    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-sysconfig\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632213    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-host\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632258    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-tuned\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632308    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-systemd-units\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632346    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632375    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-kubernetes\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632399    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-tmp\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632426    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632485    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-etc-selinux\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632515    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8e00f00-647c-463b-b317-aa670bde2fd4-serviceca\") pod \"node-ca-2n89j\" (UID: \"e8e00f00-647c-463b-b317-aa670bde2fd4\") " pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.632707 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632571    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5sw4\" (UniqueName: \"kubernetes.io/projected/d00bb862-d044-4f60-8f8e-75f3e539d7a9-kube-api-access-g5sw4\") pod \"iptables-alerter-zlmdr\" (UID: \"d00bb862-d044-4f60-8f8e-75f3e539d7a9\") " pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.633295 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632597    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lqgc\" (UniqueName: \"kubernetes.io/projected/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-kube-api-access-7lqgc\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.633295 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632620    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-var-lib-openvswitch\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.633295 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632656    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041-hosts-file\") pod \"node-resolver-ghsp2\" (UID: \"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041\") " pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.633295 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632680    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-cni-binary-copy\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.633295 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632709    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-run-multus-certs\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.633295 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632740    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-sys\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.633295 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632763    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-device-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.633295 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.632787    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74z9\" (UniqueName: \"kubernetes.io/projected/c81cee79-3ed2-4f7c-86e4-902dba35b94e-kube-api-access-d74z9\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.657894 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.657862    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 10:45:22 +0000 UTC" deadline="2027-10-23 06:33:39.022110499 +0000 UTC"
May 29 10:50:23.657894 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.657889    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12283h43m15.364224071s"
May 29 10:50:23.720501 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.720475    2547 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 29 10:50:23.733433 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733402    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.733595 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733441    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-run-openvswitch\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.733595 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733469    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2lcb9\" (UniqueName: \"kubernetes.io/projected/e8e00f00-647c-463b-b317-aa670bde2fd4-kube-api-access-2lcb9\") pod \"node-ca-2n89j\" (UID: \"e8e00f00-647c-463b-b317-aa670bde2fd4\") " pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.733595 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733491    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-daemon-config\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.733595 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733513    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-systemd\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.733595 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733529    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-run-openvswitch\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.733595 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733533    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-lib-modules\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.733595 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733587    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a152d23c-60e9-432e-98c3-799273b117fd-ovn-node-metrics-cert\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733613    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-system-cni-dir\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733639    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-cni-dir\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733659    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-lib-modules\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733661    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-modprobe-d\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733688    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-system-cni-dir\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733714    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-system-cni-dir\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733611    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-systemd\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733716    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733764    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6-agent-certs\") pod \"konnectivity-agent-nxddb\" (UID: \"a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6\") " pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733788    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6-konnectivity-ca\") pod \"konnectivity-agent-nxddb\" (UID: \"a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6\") " pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733813    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcbn\" (UniqueName: \"kubernetes.io/projected/7883155e-3d26-41dc-8c39-a954501459b5-kube-api-access-zmcbn\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733837    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-run-systemd\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733872    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-cni-netd\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.733921 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.733897    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a152d23c-60e9-432e-98c3-799273b117fd-ovnkube-config\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734106    2547 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734125    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-run-systemd\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734140    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-daemon-config\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734230    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-modprobe-d\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734232    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-cni-dir\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734283    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-registration-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734305    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-cni-netd\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734339    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-registration-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734309    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-sys-fs\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734350    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-sys-fs\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734367    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d00bb862-d044-4f60-8f8e-75f3e539d7a9-host-slash\") pod \"iptables-alerter-zlmdr\" (UID: \"d00bb862-d044-4f60-8f8e-75f3e539d7a9\") " pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734389    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041-tmp-dir\") pod \"node-resolver-ghsp2\" (UID: \"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041\") " pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734404    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-var-lib-cni-bin\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734418    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-socket-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734417    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6-konnectivity-ca\") pod \"konnectivity-agent-nxddb\" (UID: \"a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6\") " pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734438    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-sysctl-conf\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734463    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-run\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.734499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734478    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-var-lib-cni-bin\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734493    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59mz\" (UniqueName: \"kubernetes.io/projected/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-kube-api-access-t59mz\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734529    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-run\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734549    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d00bb862-d044-4f60-8f8e-75f3e539d7a9-host-slash\") pod \"iptables-alerter-zlmdr\" (UID: \"d00bb862-d044-4f60-8f8e-75f3e539d7a9\") " pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734562    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-log-socket\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734590    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a152d23c-60e9-432e-98c3-799273b117fd-ovnkube-script-lib\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734600    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a152d23c-60e9-432e-98c3-799273b117fd-ovnkube-config\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734602    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-socket-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734622    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-sysctl-conf\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734616    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-socket-dir-parent\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734656    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-socket-dir-parent\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734663    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a152d23c-60e9-432e-98c3-799273b117fd-env-overrides\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734687    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-run-k8s-cni-cncf-io\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734709    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-etc-kubernetes\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734732    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-cnibin\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734749    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041-tmp-dir\") pod \"node-resolver-ghsp2\" (UID: \"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041\") " pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734769    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-run-k8s-cni-cncf-io\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.735340 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734754    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-slash\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734788    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-slash\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734816    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-cni-bin\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734838    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-log-socket\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734846    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-etc-kubernetes\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734891    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-var-lib-cni-multus\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734914    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-cni-bin\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734951    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44zr5\" (UniqueName: \"kubernetes.io/projected/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-kube-api-access-44zr5\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.734985    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-kubelet\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735029    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-67r68\" (UniqueName: \"kubernetes.io/projected/a152d23c-60e9-432e-98c3-799273b117fd-kube-api-access-67r68\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735051    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-var-lib-cni-multus\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735056    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zttnn\" (UniqueName: \"kubernetes.io/projected/ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041-kube-api-access-zttnn\") pod \"node-resolver-ghsp2\" (UID: \"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041\") " pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735110    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a152d23c-60e9-432e-98c3-799273b117fd-env-overrides\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735155    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a152d23c-60e9-432e-98c3-799273b117fd-ovnkube-script-lib\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735217    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-run-netns\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735266    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-etc-openvswitch\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735312    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-run-netns\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735320    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-etc-openvswitch\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736034 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735345    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735354    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-kubelet\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735410    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735479    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d00bb862-d044-4f60-8f8e-75f3e539d7a9-iptables-alerter-script\") pod \"iptables-alerter-zlmdr\" (UID: \"d00bb862-d044-4f60-8f8e-75f3e539d7a9\") " pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735511    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-var-lib-kubelet\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735536    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-run-ovn\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735561    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-os-release\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735572    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-var-lib-kubelet\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735586    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-conf-dir\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735613    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-var-lib-kubelet\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735618    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-run-ovn\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735660    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-multus-conf-dir\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735659    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735664    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-os-release\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735695    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-node-log\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735707    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-var-lib-kubelet\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735722    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-run-netns\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735734    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-node-log\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.736835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735747    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-hostroot\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735803    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-run-netns\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735833    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-sysconfig\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735857    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-host\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735898    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-hostroot\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735920    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d00bb862-d044-4f60-8f8e-75f3e539d7a9-iptables-alerter-script\") pod \"iptables-alerter-zlmdr\" (UID: \"d00bb862-d044-4f60-8f8e-75f3e539d7a9\") " pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735927    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-host\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735932    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-tuned\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735965    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.735968    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-sysconfig\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736012    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-systemd-units\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736051    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736081    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-kubernetes\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736083    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-systemd-units\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736106    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-tmp\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736134    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-os-release\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736145    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.737686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736170    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-kubernetes\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736193    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736265    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-etc-selinux\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736291    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8e00f00-647c-463b-b317-aa670bde2fd4-serviceca\") pod \"node-ca-2n89j\" (UID: \"e8e00f00-647c-463b-b317-aa670bde2fd4\") " pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736297    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736316    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g5sw4\" (UniqueName: \"kubernetes.io/projected/d00bb862-d044-4f60-8f8e-75f3e539d7a9-kube-api-access-g5sw4\") pod \"iptables-alerter-zlmdr\" (UID: \"d00bb862-d044-4f60-8f8e-75f3e539d7a9\") " pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736371    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqgc\" (UniqueName: \"kubernetes.io/projected/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-kube-api-access-7lqgc\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736389    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-etc-selinux\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736398    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736449    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-var-lib-openvswitch\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736474    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041-hosts-file\") pod \"node-resolver-ghsp2\" (UID: \"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041\") " pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736516    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-cni-binary-copy\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736542    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-run-multus-certs\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736582    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-sys\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736614    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736626    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a152d23c-60e9-432e-98c3-799273b117fd-var-lib-openvswitch\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736659    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-device-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.738441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736690    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d74z9\" (UniqueName: \"kubernetes.io/projected/c81cee79-3ed2-4f7c-86e4-902dba35b94e-kube-api-access-d74z9\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736749    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8e00f00-647c-463b-b317-aa670bde2fd4-serviceca\") pod \"node-ca-2n89j\" (UID: \"e8e00f00-647c-463b-b317-aa670bde2fd4\") " pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736823    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-sys\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736859    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c81cee79-3ed2-4f7c-86e4-902dba35b94e-device-dir\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736876    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041-hosts-file\") pod \"node-resolver-ghsp2\" (UID: \"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041\") " pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736921    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-host-run-multus-certs\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.737219    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8e00f00-647c-463b-b317-aa670bde2fd4-host\") pod \"node-ca-2n89j\" (UID: \"e8e00f00-647c-463b-b317-aa670bde2fd4\") " pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.736717    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8e00f00-647c-463b-b317-aa670bde2fd4-host\") pod \"node-ca-2n89j\" (UID: \"e8e00f00-647c-463b-b317-aa670bde2fd4\") " pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.737292    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-cnibin\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.737341    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-sysctl-d\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.737352    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-cni-binary-copy\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.737425    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-cnibin\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.737499    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-sysctl-d\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.737939    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a152d23c-60e9-432e-98c3-799273b117fd-ovn-node-metrics-cert\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.738103    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6-agent-certs\") pod \"konnectivity-agent-nxddb\" (UID: \"a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6\") " pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.738381    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-etc-tuned\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.739053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.738457    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-tmp\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.741526 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.741498    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lcb9\" (UniqueName: \"kubernetes.io/projected/e8e00f00-647c-463b-b317-aa670bde2fd4-kube-api-access-2lcb9\") pod \"node-ca-2n89j\" (UID: \"e8e00f00-647c-463b-b317-aa670bde2fd4\") " pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.742562 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.742539    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttnn\" (UniqueName: \"kubernetes.io/projected/ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041-kube-api-access-zttnn\") pod \"node-resolver-ghsp2\" (UID: \"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041\") " pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.742968 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.742926    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44zr5\" (UniqueName: \"kubernetes.io/projected/4dcaa3dd-8715-4b80-84e1-5000e413b7e5-kube-api-access-44zr5\") pod \"multus-97zrn\" (UID: \"4dcaa3dd-8715-4b80-84e1-5000e413b7e5\") " pod="openshift-multus/multus-97zrn"
May 29 10:50:23.743250 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.743218    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r68\" (UniqueName: \"kubernetes.io/projected/a152d23c-60e9-432e-98c3-799273b117fd-kube-api-access-67r68\") pod \"ovnkube-node-c8682\" (UID: \"a152d23c-60e9-432e-98c3-799273b117fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.744314 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.744231    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74z9\" (UniqueName: \"kubernetes.io/projected/c81cee79-3ed2-4f7c-86e4-902dba35b94e-kube-api-access-d74z9\") pod \"aws-ebs-csi-driver-node-sg9w6\" (UID: \"c81cee79-3ed2-4f7c-86e4-902dba35b94e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.744314 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.744281    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lqgc\" (UniqueName: \"kubernetes.io/projected/d18acbe7-c88b-44c1-9f3e-a0c32828dbb8-kube-api-access-7lqgc\") pod \"tuned-x84w9\" (UID: \"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8\") " pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.744463 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.744439    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5sw4\" (UniqueName: \"kubernetes.io/projected/d00bb862-d044-4f60-8f8e-75f3e539d7a9-kube-api-access-g5sw4\") pod \"iptables-alerter-zlmdr\" (UID: \"d00bb862-d044-4f60-8f8e-75f3e539d7a9\") " pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.838358 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838281    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838358 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838334    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:23.838549 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838365    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-os-release\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838549 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838394    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838549 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838426    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:23.838549 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838472    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838549 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838508    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-system-cni-dir\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838549 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838517    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-os-release\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838549 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838532    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838960 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838560    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcbn\" (UniqueName: \"kubernetes.io/projected/7883155e-3d26-41dc-8c39-a954501459b5-kube-api-access-zmcbn\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:23.838960 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838594    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t59mz\" (UniqueName: \"kubernetes.io/projected/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-kube-api-access-t59mz\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838960 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838621    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-cnibin\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838960 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:23.838640    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:23.838960 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838700    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-cnibin\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.838960 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:23.838758    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs podName:7883155e-3d26-41dc-8c39-a954501459b5 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:24.338700602 +0000 UTC m=+3.215262816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs") pod "network-metrics-daemon-fqncf" (UID: "7883155e-3d26-41dc-8c39-a954501459b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:23.839369 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838976    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.839369 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.838986    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.839369 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.839099    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-system-cni-dir\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.839369 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.839119    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.839369 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.839341    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.846356 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:23.846333    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:50:23.846356 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:23.846357    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:50:23.846523 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:23.846371    2547 projected.go:194] Error preparing data for projected volume kube-api-access-64gfw for pod openshift-network-diagnostics/network-check-target-4524p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:23.846523 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:23.846447    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw podName:532d0df8-fdbb-40c5-8f95-d6f2975fc47e nodeName:}" failed. No retries permitted until 2026-05-29 10:50:24.346428926 +0000 UTC m=+3.222991133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-64gfw" (UniqueName: "kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw") pod "network-check-target-4524p" (UID: "532d0df8-fdbb-40c5-8f95-d6f2975fc47e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:23.848027 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.848005    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcbn\" (UniqueName: \"kubernetes.io/projected/7883155e-3d26-41dc-8c39-a954501459b5-kube-api-access-zmcbn\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:23.848773 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.848755    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59mz\" (UniqueName: \"kubernetes.io/projected/c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7-kube-api-access-t59mz\") pod \"multus-additional-cni-plugins-qm79t\" (UID: \"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7\") " pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:23.901074 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.901040    2547 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 10:50:23.917002 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.916970    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:23.924942 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.924917    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6"
May 29 10:50:23.934357 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.934327    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x84w9"
May 29 10:50:23.938976 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.938954    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2n89j"
May 29 10:50:23.945679 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.945661    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:23.952299 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.952275    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ghsp2"
May 29 10:50:23.958870 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.958844    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlmdr"
May 29 10:50:23.966457 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.966434    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-97zrn"
May 29 10:50:23.972059 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:23.972044    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qm79t"
May 29 10:50:24.341558 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.341532    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:24.341750 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:24.341685    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:24.341794 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:24.341755    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs podName:7883155e-3d26-41dc-8c39-a954501459b5 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:25.341733137 +0000 UTC m=+4.218295344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs") pod "network-metrics-daemon-fqncf" (UID: "7883155e-3d26-41dc-8c39-a954501459b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:24.343505 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:24.343474    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14ee7fa_f840_4bc7_b2a5_28e8d5037fe7.slice/crio-ec1c4ecfd604cb64c2203b6fb90f6e92c4380b2c8bf9d96a2ea30425a0050864 WatchSource:0}: Error finding container ec1c4ecfd604cb64c2203b6fb90f6e92c4380b2c8bf9d96a2ea30425a0050864: Status 404 returned error can't find the container with id ec1c4ecfd604cb64c2203b6fb90f6e92c4380b2c8bf9d96a2ea30425a0050864
May 29 10:50:24.344507 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:24.344480    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2f5640_3ce2_4e84_acbb_f3f8b6ed0041.slice/crio-5587cf1b0a27423ee42d136f77a77475d7961a42538d442c11b61ccd3a92b92d WatchSource:0}: Error finding container 5587cf1b0a27423ee42d136f77a77475d7961a42538d442c11b61ccd3a92b92d: Status 404 returned error can't find the container with id 5587cf1b0a27423ee42d136f77a77475d7961a42538d442c11b61ccd3a92b92d
May 29 10:50:24.345422 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:24.345397    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcaa3dd_8715_4b80_84e1_5000e413b7e5.slice/crio-d0ff30171c662254f23f7904b30ced025095432816916cbe69bbac555c0093ca WatchSource:0}: Error finding container d0ff30171c662254f23f7904b30ced025095432816916cbe69bbac555c0093ca: Status 404 returned error can't find the container with id d0ff30171c662254f23f7904b30ced025095432816916cbe69bbac555c0093ca
May 29 10:50:24.348543 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:24.348526    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda152d23c_60e9_432e_98c3_799273b117fd.slice/crio-0f7ea39a6f180826a7d8c70cd93b5e727dae149ffa319523394816de3360af6a WatchSource:0}: Error finding container 0f7ea39a6f180826a7d8c70cd93b5e727dae149ffa319523394816de3360af6a: Status 404 returned error can't find the container with id 0f7ea39a6f180826a7d8c70cd93b5e727dae149ffa319523394816de3360af6a
May 29 10:50:24.349627 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:24.349590    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18acbe7_c88b_44c1_9f3e_a0c32828dbb8.slice/crio-3e673edd18ed430775977ade87b8f909cd510c97859a98b845641149374ec62c WatchSource:0}: Error finding container 3e673edd18ed430775977ade87b8f909cd510c97859a98b845641149374ec62c: Status 404 returned error can't find the container with id 3e673edd18ed430775977ade87b8f909cd510c97859a98b845641149374ec62c
May 29 10:50:24.350985 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:24.350264    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f095a4_c23f_42ed_b8c5_c76aea6f5bb6.slice/crio-ac3a32a96b1a8c8eb40e90a5fc53f0123e934880808d051682c3fe8b3deaa786 WatchSource:0}: Error finding container ac3a32a96b1a8c8eb40e90a5fc53f0123e934880808d051682c3fe8b3deaa786: Status 404 returned error can't find the container with id ac3a32a96b1a8c8eb40e90a5fc53f0123e934880808d051682c3fe8b3deaa786
May 29 10:50:24.352305 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:24.351667    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e00f00_647c_463b_b317_aa670bde2fd4.slice/crio-4eca9fdd7fab6c9734c82a09aa4106a9dfb3cf824f46e3b54d4f9704db7a3a32 WatchSource:0}: Error finding container 4eca9fdd7fab6c9734c82a09aa4106a9dfb3cf824f46e3b54d4f9704db7a3a32: Status 404 returned error can't find the container with id 4eca9fdd7fab6c9734c82a09aa4106a9dfb3cf824f46e3b54d4f9704db7a3a32
May 29 10:50:24.352629 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:24.352539    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00bb862_d044_4f60_8f8e_75f3e539d7a9.slice/crio-a82bc778c38fc8e6a7d6ef2fb294d7ed540707a586f4736eda8bbd942d7b59d1 WatchSource:0}: Error finding container a82bc778c38fc8e6a7d6ef2fb294d7ed540707a586f4736eda8bbd942d7b59d1: Status 404 returned error can't find the container with id a82bc778c38fc8e6a7d6ef2fb294d7ed540707a586f4736eda8bbd942d7b59d1
May 29 10:50:24.354158 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:24.354062    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81cee79_3ed2_4f7c_86e4_902dba35b94e.slice/crio-1bea53997bb349d407d691a295e32a0261d9aa00c21a5d0320424237c413506d WatchSource:0}: Error finding container 1bea53997bb349d407d691a295e32a0261d9aa00c21a5d0320424237c413506d: Status 404 returned error can't find the container with id 1bea53997bb349d407d691a295e32a0261d9aa00c21a5d0320424237c413506d
May 29 10:50:24.442138 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.442109    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:24.442295 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:24.442281    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:50:24.442339 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:24.442304    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:50:24.442339 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:24.442318    2547 projected.go:194] Error preparing data for projected volume kube-api-access-64gfw for pod openshift-network-diagnostics/network-check-target-4524p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:24.442434 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:24.442386    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw podName:532d0df8-fdbb-40c5-8f95-d6f2975fc47e nodeName:}" failed. No retries permitted until 2026-05-29 10:50:25.442354403 +0000 UTC m=+4.318916608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-64gfw" (UniqueName: "kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw") pod "network-check-target-4524p" (UID: "532d0df8-fdbb-40c5-8f95-d6f2975fc47e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:24.658213 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.658172    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 10:45:22 +0000 UTC" deadline="2028-01-25 17:11:40.600717373 +0000 UTC"
May 29 10:50:24.658213 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.658212    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14550h21m15.942508589s"
May 29 10:50:24.722096 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.722056    2547 generic.go:358] "Generic (PLEG): container finished" podID="6c7eed8448233ff397c88b5c98ff0229" containerID="95255ea065359cd315e26589a7d622f838c73dbecad02a2a788e48675044007c" exitCode=0
May 29 10:50:24.722276 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.722147    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" event={"ID":"6c7eed8448233ff397c88b5c98ff0229","Type":"ContainerDied","Data":"95255ea065359cd315e26589a7d622f838c73dbecad02a2a788e48675044007c"}
May 29 10:50:24.724300 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.724267    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlmdr" event={"ID":"d00bb862-d044-4f60-8f8e-75f3e539d7a9","Type":"ContainerStarted","Data":"a82bc778c38fc8e6a7d6ef2fb294d7ed540707a586f4736eda8bbd942d7b59d1"}
May 29 10:50:24.726742 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.726706    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2n89j" event={"ID":"e8e00f00-647c-463b-b317-aa670bde2fd4","Type":"ContainerStarted","Data":"4eca9fdd7fab6c9734c82a09aa4106a9dfb3cf824f46e3b54d4f9704db7a3a32"}
May 29 10:50:24.727758 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.727733    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nxddb" event={"ID":"a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6","Type":"ContainerStarted","Data":"ac3a32a96b1a8c8eb40e90a5fc53f0123e934880808d051682c3fe8b3deaa786"}
May 29 10:50:24.728910 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.728789    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x84w9" event={"ID":"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8","Type":"ContainerStarted","Data":"3e673edd18ed430775977ade87b8f909cd510c97859a98b845641149374ec62c"}
May 29 10:50:24.729969 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.729937    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-97zrn" event={"ID":"4dcaa3dd-8715-4b80-84e1-5000e413b7e5","Type":"ContainerStarted","Data":"d0ff30171c662254f23f7904b30ced025095432816916cbe69bbac555c0093ca"}
May 29 10:50:24.731278 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.731110    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm79t" event={"ID":"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7","Type":"ContainerStarted","Data":"ec1c4ecfd604cb64c2203b6fb90f6e92c4380b2c8bf9d96a2ea30425a0050864"}
May 29 10:50:24.732860 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.732839    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" event={"ID":"f693add312221a196297edaec6e814ca","Type":"ContainerStarted","Data":"e06e4e66eff477217151ef29d6d56bda12f654204b489b94f48950b226d67970"}
May 29 10:50:24.734119 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.734095    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6" event={"ID":"c81cee79-3ed2-4f7c-86e4-902dba35b94e","Type":"ContainerStarted","Data":"1bea53997bb349d407d691a295e32a0261d9aa00c21a5d0320424237c413506d"}
May 29 10:50:24.735689 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.735666    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerStarted","Data":"0f7ea39a6f180826a7d8c70cd93b5e727dae149ffa319523394816de3360af6a"}
May 29 10:50:24.736981 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:24.736956    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ghsp2" event={"ID":"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041","Type":"ContainerStarted","Data":"5587cf1b0a27423ee42d136f77a77475d7961a42538d442c11b61ccd3a92b92d"}
May 29 10:50:25.349163 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:25.349072    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:25.349365 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:25.349283    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:25.349365 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:25.349361    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs podName:7883155e-3d26-41dc-8c39-a954501459b5 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:27.349340427 +0000 UTC m=+6.225902637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs") pod "network-metrics-daemon-fqncf" (UID: "7883155e-3d26-41dc-8c39-a954501459b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:25.450492 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:25.450449    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:25.450685 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:25.450614    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:50:25.450685 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:25.450635    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:50:25.450685 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:25.450648    2547 projected.go:194] Error preparing data for projected volume kube-api-access-64gfw for pod openshift-network-diagnostics/network-check-target-4524p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:25.450840 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:25.450705    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw podName:532d0df8-fdbb-40c5-8f95-d6f2975fc47e nodeName:}" failed. No retries permitted until 2026-05-29 10:50:27.450686889 +0000 UTC m=+6.327249109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-64gfw" (UniqueName: "kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw") pod "network-check-target-4524p" (UID: "532d0df8-fdbb-40c5-8f95-d6f2975fc47e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:25.714261 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:25.713440    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:25.714261 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:25.713581    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:25.714261 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:25.714031    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:25.714261 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:25.714140    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:25.750419 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:25.750346    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" event={"ID":"6c7eed8448233ff397c88b5c98ff0229","Type":"ContainerStarted","Data":"bc019fbb920f8b5677a734652134eea6fdc30a04bc9973f25be548ddb0ddabdb"}
May 29 10:50:25.765255 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:25.765186    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" podStartSLOduration=2.765166211 podStartE2EDuration="2.765166211s" podCreationTimestamp="2026-05-29 10:50:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:50:24.745544884 +0000 UTC m=+3.622107113" watchObservedRunningTime="2026-05-29 10:50:25.765166211 +0000 UTC m=+4.641728441"
May 29 10:50:26.015185 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.015072    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" podStartSLOduration=3.015050551 podStartE2EDuration="3.015050551s" podCreationTimestamp="2026-05-29 10:50:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:50:25.765549744 +0000 UTC m=+4.642111977" watchObservedRunningTime="2026-05-29 10:50:26.015050551 +0000 UTC m=+4.891612780"
May 29 10:50:26.015847 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.015822    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-v8pkg"]
May 29 10:50:26.024107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.023771    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.024107 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:26.023907    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:26.055219 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.055012    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a07d3f20-9317-46cb-aab7-097ac29c23a3-kubelet-config\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.055219 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.055061    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a07d3f20-9317-46cb-aab7-097ac29c23a3-dbus\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.055219 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.055132    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.157211 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.156467    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.157211 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.156538    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a07d3f20-9317-46cb-aab7-097ac29c23a3-kubelet-config\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.157211 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.156566    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a07d3f20-9317-46cb-aab7-097ac29c23a3-dbus\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.157211 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.156740    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a07d3f20-9317-46cb-aab7-097ac29c23a3-dbus\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.157211 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:26.156849    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 10:50:26.157211 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:26.156913    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret podName:a07d3f20-9317-46cb-aab7-097ac29c23a3 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:26.656892924 +0000 UTC m=+5.533455132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret") pod "global-pull-secret-syncer-v8pkg" (UID: "a07d3f20-9317-46cb-aab7-097ac29c23a3") : object "kube-system"/"original-pull-secret" not registered
May 29 10:50:26.157211 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.157159    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a07d3f20-9317-46cb-aab7-097ac29c23a3-kubelet-config\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.660217 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:26.660092    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:26.660432 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:26.660254    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 10:50:26.660432 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:26.660321    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret podName:a07d3f20-9317-46cb-aab7-097ac29c23a3 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:27.660301437 +0000 UTC m=+6.536863648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret") pod "global-pull-secret-syncer-v8pkg" (UID: "a07d3f20-9317-46cb-aab7-097ac29c23a3") : object "kube-system"/"original-pull-secret" not registered
May 29 10:50:27.365804 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:27.365762    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:27.366280 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.365959    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:27.366280 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.366028    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs podName:7883155e-3d26-41dc-8c39-a954501459b5 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:31.36600733 +0000 UTC m=+10.242569559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs") pod "network-metrics-daemon-fqncf" (UID: "7883155e-3d26-41dc-8c39-a954501459b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:27.466809 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:27.466760    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:27.467005 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.466950    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:50:27.467005 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.466970    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:50:27.467005 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.466982    2547 projected.go:194] Error preparing data for projected volume kube-api-access-64gfw for pod openshift-network-diagnostics/network-check-target-4524p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:27.467164 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.467040    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw podName:532d0df8-fdbb-40c5-8f95-d6f2975fc47e nodeName:}" failed. No retries permitted until 2026-05-29 10:50:31.467021961 +0000 UTC m=+10.343584182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-64gfw" (UniqueName: "kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw") pod "network-check-target-4524p" (UID: "532d0df8-fdbb-40c5-8f95-d6f2975fc47e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:27.668966 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:27.668861    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:27.669127 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.669050    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 10:50:27.669127 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.669122    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret podName:a07d3f20-9317-46cb-aab7-097ac29c23a3 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:29.669102745 +0000 UTC m=+8.545664952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret") pod "global-pull-secret-syncer-v8pkg" (UID: "a07d3f20-9317-46cb-aab7-097ac29c23a3") : object "kube-system"/"original-pull-secret" not registered
May 29 10:50:27.713828 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:27.713795    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:27.713994 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.713919    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:27.713994 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:27.713943    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:27.714110 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.714062    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:27.714110 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:27.713804    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:27.714200 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:27.714154    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:29.687398 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:29.686867    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:29.687398 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:29.687031    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 10:50:29.687398 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:29.687095    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret podName:a07d3f20-9317-46cb-aab7-097ac29c23a3 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:33.687073662 +0000 UTC m=+12.563635881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret") pod "global-pull-secret-syncer-v8pkg" (UID: "a07d3f20-9317-46cb-aab7-097ac29c23a3") : object "kube-system"/"original-pull-secret" not registered
May 29 10:50:29.713830 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:29.713352    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:29.713830 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:29.713478    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:29.713830 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:29.713699    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:29.713830 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:29.713821    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:29.716184 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:29.714917    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:29.716184 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:29.715024    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:31.403948 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:31.403907    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:31.404430 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:31.404087    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:31.404430 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:31.404153    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs podName:7883155e-3d26-41dc-8c39-a954501459b5 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:39.404136211 +0000 UTC m=+18.280698427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs") pod "network-metrics-daemon-fqncf" (UID: "7883155e-3d26-41dc-8c39-a954501459b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:31.504901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:31.504861    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:31.505066 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:31.505038    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:50:31.505066 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:31.505061    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:50:31.505163 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:31.505076    2547 projected.go:194] Error preparing data for projected volume kube-api-access-64gfw for pod openshift-network-diagnostics/network-check-target-4524p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:31.505163 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:31.505138    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw podName:532d0df8-fdbb-40c5-8f95-d6f2975fc47e nodeName:}" failed. No retries permitted until 2026-05-29 10:50:39.50512006 +0000 UTC m=+18.381682284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-64gfw" (UniqueName: "kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw") pod "network-check-target-4524p" (UID: "532d0df8-fdbb-40c5-8f95-d6f2975fc47e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:31.713564 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:31.713305    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:31.713564 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:31.713388    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:31.713564 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:31.713557    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:31.713834 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:31.713592    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:31.713834 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:31.713716    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:31.713909 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:31.713873    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:33.712850 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:33.712812    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:33.712850 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:33.712812    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:33.713409 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:33.712943    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:33.713409 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:33.712956    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:33.713409 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:33.713065    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:33.713409 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:33.713170    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:33.719869 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:33.719842    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:33.720017 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:33.719955    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 10:50:33.720017 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:33.720011    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret podName:a07d3f20-9317-46cb-aab7-097ac29c23a3 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:41.719994828 +0000 UTC m=+20.596557045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret") pod "global-pull-secret-syncer-v8pkg" (UID: "a07d3f20-9317-46cb-aab7-097ac29c23a3") : object "kube-system"/"original-pull-secret" not registered
May 29 10:50:35.713163 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:35.713126    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:35.713726 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:35.713127    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:35.713726 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:35.713293    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:35.713726 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:35.713340    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:35.713726 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:35.713128    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:35.713726 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:35.713422    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:37.712401 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:37.712366    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:37.712829 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:37.712366    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:37.712829 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:37.712499    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:37.712829 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:37.712366    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:37.712829 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:37.712589    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:37.712829 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:37.712728    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:39.463357 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:39.463317    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:39.463775 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:39.463476    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:39.463775 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:39.463555    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs podName:7883155e-3d26-41dc-8c39-a954501459b5 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:55.463538746 +0000 UTC m=+34.340100956 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs") pod "network-metrics-daemon-fqncf" (UID: "7883155e-3d26-41dc-8c39-a954501459b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:39.564541 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:39.564497    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:39.564701 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:39.564678    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:50:39.564759 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:39.564706    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:50:39.564759 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:39.564718    2547 projected.go:194] Error preparing data for projected volume kube-api-access-64gfw for pod openshift-network-diagnostics/network-check-target-4524p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:39.564829 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:39.564780    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw podName:532d0df8-fdbb-40c5-8f95-d6f2975fc47e nodeName:}" failed. No retries permitted until 2026-05-29 10:50:55.564765666 +0000 UTC m=+34.441327875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-64gfw" (UniqueName: "kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw") pod "network-check-target-4524p" (UID: "532d0df8-fdbb-40c5-8f95-d6f2975fc47e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:39.713101 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:39.713070    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:39.713281 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:39.713073    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:39.713281 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:39.713187    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:39.713491 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:39.713285    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:39.713491 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:39.713070    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:39.713491 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:39.713373    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:41.713619 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:41.713583    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:41.714109 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:41.713688    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:41.714109 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:41.713740    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:41.714109 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:41.713763    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:41.714109 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:41.713858    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:41.714109 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:41.713956    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:41.783035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:41.783004    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:41.783196 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:41.783132    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 10:50:41.783196 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:41.783196    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret podName:a07d3f20-9317-46cb-aab7-097ac29c23a3 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:57.783178763 +0000 UTC m=+36.659740969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret") pod "global-pull-secret-syncer-v8pkg" (UID: "a07d3f20-9317-46cb-aab7-097ac29c23a3") : object "kube-system"/"original-pull-secret" not registered
May 29 10:50:42.782723 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.782522    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2n89j" event={"ID":"e8e00f00-647c-463b-b317-aa670bde2fd4","Type":"ContainerStarted","Data":"05fa443a8e2dd90859380328fea86f5542b12622a3cf89a25719e006aba9acd4"}
May 29 10:50:42.784032 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.784001    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nxddb" event={"ID":"a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6","Type":"ContainerStarted","Data":"ee76f310770453d5c3de5e25c8936c95df2959c39028cb90f25d5ec8b78314f5"}
May 29 10:50:42.785426 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.785403    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x84w9" event={"ID":"d18acbe7-c88b-44c1-9f3e-a0c32828dbb8","Type":"ContainerStarted","Data":"7702e0ba311a1cd5520b031323847bf877b74b34ae959546954a5feb18c4279b"}
May 29 10:50:42.786710 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.786658    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-97zrn" event={"ID":"4dcaa3dd-8715-4b80-84e1-5000e413b7e5","Type":"ContainerStarted","Data":"bcf6a0111dde0cb27737acce62f032ad6a8cfc263148e351ac6def69f45f43d4"}
May 29 10:50:42.787981 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.787960    2547 generic.go:358] "Generic (PLEG): container finished" podID="c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7" containerID="b482854a9bd235ab41350f7ff29fc1e237b179905391d46822ee96aa0cb80853" exitCode=0
May 29 10:50:42.788090 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.788032    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm79t" event={"ID":"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7","Type":"ContainerDied","Data":"b482854a9bd235ab41350f7ff29fc1e237b179905391d46822ee96aa0cb80853"}
May 29 10:50:42.789460 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.789355    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6" event={"ID":"c81cee79-3ed2-4f7c-86e4-902dba35b94e","Type":"ContainerStarted","Data":"71da95dffd05283f19ec70b85f3aa329e23e8c48fa53d45a02f2a5ab5440fa8b"}
May 29 10:50:42.791123 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.791108    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 10:50:42.791467 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.791447    2547 generic.go:358] "Generic (PLEG): container finished" podID="a152d23c-60e9-432e-98c3-799273b117fd" containerID="5b404c076915175d8d35010c19e09c360d6dae246aab29e101975cc795883bc0" exitCode=1
May 29 10:50:42.791561 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.791505    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerStarted","Data":"382bf2129f8f5e9d7981506c78ed33d179ceeb7a88a47bc326d6dc5bcbaa7c91"}
May 29 10:50:42.791561 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.791533    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerDied","Data":"5b404c076915175d8d35010c19e09c360d6dae246aab29e101975cc795883bc0"}
May 29 10:50:42.791561 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.791550    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerStarted","Data":"83df7b92bbef0731108b83c65ab6c724fa7b407fdd9e18b86500f6c6b9fde01f"}
May 29 10:50:42.792738 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.792716    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ghsp2" event={"ID":"ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041","Type":"ContainerStarted","Data":"aeefe1959bd92ab7fa95a00153c1470e8c81640f7a0d9286087917310689caae"}
May 29 10:50:42.795567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.795520    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2n89j" podStartSLOduration=11.902732272 podStartE2EDuration="21.795505876s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:50:24.354556234 +0000 UTC m=+3.231118444" lastFinishedPulling="2026-05-29 10:50:34.247329828 +0000 UTC m=+13.123892048" observedRunningTime="2026-05-29 10:50:42.79528255 +0000 UTC m=+21.671844777" watchObservedRunningTime="2026-05-29 10:50:42.795505876 +0000 UTC m=+21.672068110"
May 29 10:50:42.825474 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.825427    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-ghsp2" podStartSLOduration=3.934083906 podStartE2EDuration="21.825412038s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:50:24.346394409 +0000 UTC m=+3.222956629" lastFinishedPulling="2026-05-29 10:50:42.237722556 +0000 UTC m=+21.114284761" observedRunningTime="2026-05-29 10:50:42.808696663 +0000 UTC m=+21.685258892" watchObservedRunningTime="2026-05-29 10:50:42.825412038 +0000 UTC m=+21.701974269"
May 29 10:50:42.825589 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.825538    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-97zrn" podStartSLOduration=3.858923866 podStartE2EDuration="21.825531987s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:50:24.348029965 +0000 UTC m=+3.224592173" lastFinishedPulling="2026-05-29 10:50:42.314638066 +0000 UTC m=+21.191200294" observedRunningTime="2026-05-29 10:50:42.825313743 +0000 UTC m=+21.701875971" watchObservedRunningTime="2026-05-29 10:50:42.825531987 +0000 UTC m=+21.702094215"
May 29 10:50:42.859395 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.859175    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nxddb" podStartSLOduration=3.973199879 podStartE2EDuration="21.859159305s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:50:24.352208072 +0000 UTC m=+3.228770277" lastFinishedPulling="2026-05-29 10:50:42.238167483 +0000 UTC m=+21.114729703" observedRunningTime="2026-05-29 10:50:42.858501319 +0000 UTC m=+21.735063552" watchObservedRunningTime="2026-05-29 10:50:42.859159305 +0000 UTC m=+21.735721540"
May 29 10:50:42.876540 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:42.876486    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x84w9" podStartSLOduration=3.990642218 podStartE2EDuration="21.876466489s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:50:24.351937875 +0000 UTC m=+3.228500087" lastFinishedPulling="2026-05-29 10:50:42.237762146 +0000 UTC m=+21.114324358" observedRunningTime="2026-05-29 10:50:42.876159437 +0000 UTC m=+21.752721668" watchObservedRunningTime="2026-05-29 10:50:42.876466489 +0000 UTC m=+21.753028718"
May 29 10:50:43.712682 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.712650    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:43.712682 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.712677    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:43.712917 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.712687    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:43.712917 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:43.712800    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:43.712990 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:43.712956    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:43.713105 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:43.713065    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:43.796474 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.796425    2547 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 29 10:50:43.796965 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.796897    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6" event={"ID":"c81cee79-3ed2-4f7c-86e4-902dba35b94e","Type":"ContainerStarted","Data":"7f541f7a50f493fe2c4802022b63d754d2a020edd763fa476bd232dd156616b9"}
May 29 10:50:43.799880 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.799845    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 10:50:43.800274 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.800224    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerStarted","Data":"d1a5a6819772e695c6a115a711069caaa482e0c1a2fe8ef6e6dd12265ce3d26d"}
May 29 10:50:43.800378 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.800276    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerStarted","Data":"d3a4b0a0f6a833f0720007ea2eb46bad1c2f6868c949ec9c9e61f02ba0d5026a"}
May 29 10:50:43.800378 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.800292    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerStarted","Data":"4047f3fe7b859522c1af0cd826529246e51e768ac20234d39376a011054f42ec"}
May 29 10:50:43.801733 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.801678    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlmdr" event={"ID":"d00bb862-d044-4f60-8f8e-75f3e539d7a9","Type":"ContainerStarted","Data":"1a79bbeb3e198f8e9934e87c63c2ffe75e5d60ed46353ff132ab8754abdbcdbf"}
May 29 10:50:43.816953 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:43.816911    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zlmdr" podStartSLOduration=4.934741763 podStartE2EDuration="22.816894737s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:50:24.355934743 +0000 UTC m=+3.232496963" lastFinishedPulling="2026-05-29 10:50:42.238087726 +0000 UTC m=+21.114649937" observedRunningTime="2026-05-29 10:50:43.816635782 +0000 UTC m=+22.693198010" watchObservedRunningTime="2026-05-29 10:50:43.816894737 +0000 UTC m=+22.693456964"
May 29 10:50:44.682446 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:44.682337    2547 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-29T10:50:43.796446674Z","UUID":"6da20305-9b9f-46d7-a683-a80250fa0e11","Handler":null,"Name":"","Endpoint":""}
May 29 10:50:44.684118 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:44.684094    2547 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 29 10:50:44.684262 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:44.684127    2547 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 29 10:50:45.712691 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:45.712511    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:45.713136 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:45.712511    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:45.713136 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:45.712772    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:45.713136 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:45.712856    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:45.713136 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:45.712515    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:45.713136 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:45.712965    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:45.807606 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:45.807562    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6" event={"ID":"c81cee79-3ed2-4f7c-86e4-902dba35b94e","Type":"ContainerStarted","Data":"6c26eb2bf07afa63bc26f9ebf600d6b3c91e6b4ddac1d8eec30a28fe5a07444d"}
May 29 10:50:45.810650 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:45.810618    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 10:50:45.811117 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:45.811086    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerStarted","Data":"d7b825a6d5acb358d48252e229ad2d99341afacf90b270709eecb0c8152825ec"}
May 29 10:50:45.823953 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:45.823902    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg9w6" podStartSLOduration=4.207053409 podStartE2EDuration="24.82389073s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:50:24.359524912 +0000 UTC m=+3.236087118" lastFinishedPulling="2026-05-29 10:50:44.976362218 +0000 UTC m=+23.852924439" observedRunningTime="2026-05-29 10:50:45.82365332 +0000 UTC m=+24.700215551" watchObservedRunningTime="2026-05-29 10:50:45.82389073 +0000 UTC m=+24.700452956"
May 29 10:50:46.147197 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:46.147112    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:46.147851 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:46.147825    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:46.813216 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:46.813182    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:46.813816 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:46.813563    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nxddb"
May 29 10:50:47.712519 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:47.712483    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:47.712519 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:47.712509    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:47.712753 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:47.712591    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:47.712753 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:47.712599    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:47.712753 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:47.712707    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:47.712893 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:47.712805    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:47.816089 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:47.816053    2547 generic.go:358] "Generic (PLEG): container finished" podID="c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7" containerID="1086794c6dcea04856dd0e13e90d3e6d7059331af3ec8a984b4950f9ea93ffda" exitCode=0
May 29 10:50:47.816825 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:47.816125    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm79t" event={"ID":"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7","Type":"ContainerDied","Data":"1086794c6dcea04856dd0e13e90d3e6d7059331af3ec8a984b4950f9ea93ffda"}
May 29 10:50:47.819193 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:47.819175    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 10:50:47.819524 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:47.819502    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerStarted","Data":"098c22c6012278ddc63961e8c54acfa5bcfc233a951afc4d92892dd4778948b2"}
May 29 10:50:47.819984 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:47.819968    2547 scope.go:117] "RemoveContainer" containerID="5b404c076915175d8d35010c19e09c360d6dae246aab29e101975cc795883bc0"
May 29 10:50:48.824835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:48.824776    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 10:50:48.825298 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:48.825123    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" event={"ID":"a152d23c-60e9-432e-98c3-799273b117fd","Type":"ContainerStarted","Data":"2f08473e0a4c404616a347a429082cd6b4538d9fd43ed1f104050ffede14cabb"}
May 29 10:50:48.825484 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:48.825452    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:48.825577 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:48.825496    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:48.825577 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:48.825511    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:48.840776 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:48.840750    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:48.840926 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:48.840836    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:50:48.853079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:48.853032    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c8682" podStartSLOduration=9.889065792 podStartE2EDuration="27.853017713s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:50:24.350687305 +0000 UTC m=+3.227249512" lastFinishedPulling="2026-05-29 10:50:42.314639216 +0000 UTC m=+21.191201433" observedRunningTime="2026-05-29 10:50:48.851557962 +0000 UTC m=+27.728120190" watchObservedRunningTime="2026-05-29 10:50:48.853017713 +0000 UTC m=+27.729579940"
May 29 10:50:49.256750 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:49.256721    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v8pkg"]
May 29 10:50:49.256897 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:49.256831    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:49.256937 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:49.256916    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:49.259709 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:49.259679    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4524p"]
May 29 10:50:49.259856 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:49.259791    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:49.259912 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:49.259872    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:49.260376 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:49.260355    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fqncf"]
May 29 10:50:49.260471 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:49.260450    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:49.260577 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:49.260552    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:49.828667 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:49.828633    2547 generic.go:358] "Generic (PLEG): container finished" podID="c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7" containerID="f7cb848607f77893a8ef8b41bfe6995147718c313226e8b25aee0c63f7272e90" exitCode=0
May 29 10:50:49.829142 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:49.828718    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm79t" event={"ID":"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7","Type":"ContainerDied","Data":"f7cb848607f77893a8ef8b41bfe6995147718c313226e8b25aee0c63f7272e90"}
May 29 10:50:50.713061 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:50.712892    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:50.713288 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:50.712892    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:50.713288 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:50.713152    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:50.713288 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:50.712892    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:50.713288 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:50.713214    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:50.713492 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:50.713296    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:51.834416 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:51.834377    2547 generic.go:358] "Generic (PLEG): container finished" podID="c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7" containerID="d39869c28caf72d3fc202eea015cf31b0eb00280038826194684f77016b49262" exitCode=0
May 29 10:50:51.835077 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:51.834452    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm79t" event={"ID":"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7","Type":"ContainerDied","Data":"d39869c28caf72d3fc202eea015cf31b0eb00280038826194684f77016b49262"}
May 29 10:50:52.712970 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:52.712938    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:52.713150 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:52.712938    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:52.713150 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:52.713064    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:52.713284 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:52.713161    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:52.713284 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:52.712940    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:52.713284 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:52.713273    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:54.712964 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:54.712927    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:54.712964 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:54.712948    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:54.713571 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:54.712970    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:54.713571 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:54.713056    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4524p" podUID="532d0df8-fdbb-40c5-8f95-d6f2975fc47e"
May 29 10:50:54.713571 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:54.713177    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8pkg" podUID="a07d3f20-9317-46cb-aab7-097ac29c23a3"
May 29 10:50:54.713571 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:54.713276    2547 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqncf" podUID="7883155e-3d26-41dc-8c39-a954501459b5"
May 29 10:50:55.488966 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:55.488925    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:55.489271 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:55.489094    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:55.489271 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:55.489179    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs podName:7883155e-3d26-41dc-8c39-a954501459b5 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:27.489156648 +0000 UTC m=+66.365718872 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs") pod "network-metrics-daemon-fqncf" (UID: "7883155e-3d26-41dc-8c39-a954501459b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 10:50:55.589889 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:55.589855    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:55.590061 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:55.590024    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 10:50:55.590061 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:55.590046    2547 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 10:50:55.590061 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:55.590056    2547 projected.go:194] Error preparing data for projected volume kube-api-access-64gfw for pod openshift-network-diagnostics/network-check-target-4524p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:55.590201 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:55.590113    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw podName:532d0df8-fdbb-40c5-8f95-d6f2975fc47e nodeName:}" failed. No retries permitted until 2026-05-29 10:51:27.590097455 +0000 UTC m=+66.466659665 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-64gfw" (UniqueName: "kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw") pod "network-check-target-4524p" (UID: "532d0df8-fdbb-40c5-8f95-d6f2975fc47e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 10:50:56.468868 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.468832    2547 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeReady"
May 29 10:50:56.469360 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.468981    2547 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 29 10:50:56.513798 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.513764    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9cb56d556-8drnj"]
May 29 10:50:56.549622 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.549582    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5dlbc"]
May 29 10:50:56.549805 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.549782    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.552212 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.552162    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 29 10:50:56.552407 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.552280    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 29 10:50:56.552675 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.552524    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5b8kn\""
May 29 10:50:56.552675 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.552585    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 29 10:50:56.567584 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.567549    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wzwmb"]
May 29 10:50:56.567728 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.567695    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:50:56.567992 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.567971    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 29 10:50:56.570220 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.570197    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 29 10:50:56.570337 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.570305    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fk5jh\""
May 29 10:50:56.570473 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.570455    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 29 10:50:56.570563 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.570547    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 29 10:50:56.587655 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.587630    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5dlbc"]
May 29 10:50:56.587785 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.587666    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9cb56d556-8drnj"]
May 29 10:50:56.587785 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.587679    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wzwmb"]
May 29 10:50:56.587785 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.587751    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.589955 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.589935    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 29 10:50:56.590069 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.589972    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 29 10:50:56.590069 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.590048    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4jnj\""
May 29 10:50:56.697086 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697055    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da26b43-5619-444b-929c-a54e16f29818-tmp-dir\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.697086 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697092    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.697331 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697110    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.697331 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697139    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6084d0f-a832-4de8-a7db-0489742556e1-ca-trust-extracted\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.697331 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697263    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-trusted-ca\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.697331 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697311    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-image-registry-private-configuration\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.697499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697348    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-bound-sa-token\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.697499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697388    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpm7\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-kube-api-access-qnpm7\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.697499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697431    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf7j7\" (UniqueName: \"kubernetes.io/projected/8da26b43-5619-444b-929c-a54e16f29818-kube-api-access-mf7j7\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.697499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697461    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-installation-pull-secrets\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.697499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697486    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:50:56.697717 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697508    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-registry-certificates\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.697717 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697594    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvjzf\" (UniqueName: \"kubernetes.io/projected/5c4e7979-7e81-4939-a895-b934805b1275-kube-api-access-zvjzf\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:50:56.697717 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.697633    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8da26b43-5619-444b-929c-a54e16f29818-config-volume\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.713174 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.713133    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:56.713352 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.713134    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:50:56.713352 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.713134    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:50:56.715946 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.715839    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cqxtm\""
May 29 10:50:56.715946 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.715842    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d5k7t\""
May 29 10:50:56.715946 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.715902    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 29 10:50:56.716161 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.715963    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 29 10:50:56.716161 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.715963    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 29 10:50:56.716161 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.716127    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 29 10:50:56.798035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.797996    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zvjzf\" (UniqueName: \"kubernetes.io/projected/5c4e7979-7e81-4939-a895-b934805b1275-kube-api-access-zvjzf\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:50:56.798225 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798051    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8da26b43-5619-444b-929c-a54e16f29818-config-volume\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.798225 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798083    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da26b43-5619-444b-929c-a54e16f29818-tmp-dir\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.798225 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798115    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.798225 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798141    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.798225 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798171    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6084d0f-a832-4de8-a7db-0489742556e1-ca-trust-extracted\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.798516 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798228    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-trusted-ca\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.798516 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798308    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-image-registry-private-configuration\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.798516 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:56.798333    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 10:50:56.798516 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:56.798397    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls podName:8da26b43-5619-444b-929c-a54e16f29818 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:57.298375693 +0000 UTC m=+36.174937913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls") pod "dns-default-wzwmb" (UID: "8da26b43-5619-444b-929c-a54e16f29818") : secret "dns-default-metrics-tls" not found
May 29 10:50:56.798516 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:56.798446    2547 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 10:50:56.798516 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:56.798466    2547 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9cb56d556-8drnj: secret "image-registry-tls" not found
May 29 10:50:56.798516 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798507    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da26b43-5619-444b-929c-a54e16f29818-tmp-dir\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.798516 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798335    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-bound-sa-token\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.798787 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:56.798525    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls podName:d6084d0f-a832-4de8-a7db-0489742556e1 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:57.298506895 +0000 UTC m=+36.175069112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls") pod "image-registry-9cb56d556-8drnj" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1") : secret "image-registry-tls" not found
May 29 10:50:56.798787 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798563    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpm7\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-kube-api-access-qnpm7\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.798787 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798603    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mf7j7\" (UniqueName: \"kubernetes.io/projected/8da26b43-5619-444b-929c-a54e16f29818-kube-api-access-mf7j7\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.798787 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798742    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-installation-pull-secrets\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.798787 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798751    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6084d0f-a832-4de8-a7db-0489742556e1-ca-trust-extracted\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.798969 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798811    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:50:56.798969 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798826    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8da26b43-5619-444b-929c-a54e16f29818-config-volume\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:56.798969 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.798846    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-registry-certificates\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.798969 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:56.798891    2547 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 10:50:56.798969 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:56.798942    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert podName:5c4e7979-7e81-4939-a895-b934805b1275 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:57.298927457 +0000 UTC m=+36.175489679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert") pod "ingress-canary-5dlbc" (UID: "5c4e7979-7e81-4939-a895-b934805b1275") : secret "canary-serving-cert" not found
May 29 10:50:56.799327 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.799302    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-registry-certificates\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.799427 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.799365    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-trusted-ca\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.803292 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.803266    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-installation-pull-secrets\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.803469 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.803266    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-image-registry-private-configuration\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.809870 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.809821    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-bound-sa-token\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.809870 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.809852    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvjzf\" (UniqueName: \"kubernetes.io/projected/5c4e7979-7e81-4939-a895-b934805b1275-kube-api-access-zvjzf\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:50:56.810046 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.809948    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpm7\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-kube-api-access-qnpm7\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:56.810351 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:56.810319    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf7j7\" (UniqueName: \"kubernetes.io/projected/8da26b43-5619-444b-929c-a54e16f29818-kube-api-access-mf7j7\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:57.303961 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:57.303923    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:50:57.304131 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:57.303981    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:57.304131 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:57.304008    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:57.304131 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:57.304067    2547 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 10:50:57.304131 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:57.304071    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 10:50:57.304131 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:57.304125    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert podName:5c4e7979-7e81-4939-a895-b934805b1275 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:58.304110079 +0000 UTC m=+37.180672299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert") pod "ingress-canary-5dlbc" (UID: "5c4e7979-7e81-4939-a895-b934805b1275") : secret "canary-serving-cert" not found
May 29 10:50:57.304131 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:57.304127    2547 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 10:50:57.304385 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:57.304141    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls podName:8da26b43-5619-444b-929c-a54e16f29818 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:58.304133956 +0000 UTC m=+37.180696162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls") pod "dns-default-wzwmb" (UID: "8da26b43-5619-444b-929c-a54e16f29818") : secret "dns-default-metrics-tls" not found
May 29 10:50:57.304385 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:57.304144    2547 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9cb56d556-8drnj: secret "image-registry-tls" not found
May 29 10:50:57.304385 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:57.304185    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls podName:d6084d0f-a832-4de8-a7db-0489742556e1 nodeName:}" failed. No retries permitted until 2026-05-29 10:50:58.304172636 +0000 UTC m=+37.180734842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls") pod "image-registry-9cb56d556-8drnj" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1") : secret "image-registry-tls" not found
May 29 10:50:57.809228 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:57.808947    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:57.812437 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:57.812375    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a07d3f20-9317-46cb-aab7-097ac29c23a3-original-pull-secret\") pod \"global-pull-secret-syncer-v8pkg\" (UID: \"a07d3f20-9317-46cb-aab7-097ac29c23a3\") " pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:57.925020 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:57.924987    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8pkg"
May 29 10:50:58.062604 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:58.062572    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v8pkg"]
May 29 10:50:58.067360 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:50:58.067317    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07d3f20_9317_46cb_aab7_097ac29c23a3.slice/crio-9b2efedaa86c5c457dbbda17c80dab4eb4321d5199ba4407db88c4c5a0ba593f WatchSource:0}: Error finding container 9b2efedaa86c5c457dbbda17c80dab4eb4321d5199ba4407db88c4c5a0ba593f: Status 404 returned error can't find the container with id 9b2efedaa86c5c457dbbda17c80dab4eb4321d5199ba4407db88c4c5a0ba593f
May 29 10:50:58.313413 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:58.313383    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:50:58.313586 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:58.313427    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:50:58.313586 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:58.313446    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:50:58.313586 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:58.313541    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 10:50:58.313714 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:58.313598    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls podName:8da26b43-5619-444b-929c-a54e16f29818 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:00.31358449 +0000 UTC m=+39.190146699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls") pod "dns-default-wzwmb" (UID: "8da26b43-5619-444b-929c-a54e16f29818") : secret "dns-default-metrics-tls" not found
May 29 10:50:58.313714 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:58.313543    2547 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 10:50:58.313714 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:58.313642    2547 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9cb56d556-8drnj: secret "image-registry-tls" not found
May 29 10:50:58.313714 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:58.313681    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls podName:d6084d0f-a832-4de8-a7db-0489742556e1 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:00.313669854 +0000 UTC m=+39.190232060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls") pod "image-registry-9cb56d556-8drnj" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1") : secret "image-registry-tls" not found
May 29 10:50:58.313714 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:58.313543    2547 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 10:50:58.313882 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:50:58.313722    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert podName:5c4e7979-7e81-4939-a895-b934805b1275 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:00.313713667 +0000 UTC m=+39.190275873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert") pod "ingress-canary-5dlbc" (UID: "5c4e7979-7e81-4939-a895-b934805b1275") : secret "canary-serving-cert" not found
May 29 10:50:58.853896 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:58.853865    2547 generic.go:358] "Generic (PLEG): container finished" podID="c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7" containerID="29d9c9056b51ef040327df7df822c50ffcb5fa433894d7499d851e1f6a07de64" exitCode=0
May 29 10:50:58.854326 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:58.853946    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm79t" event={"ID":"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7","Type":"ContainerDied","Data":"29d9c9056b51ef040327df7df822c50ffcb5fa433894d7499d851e1f6a07de64"}
May 29 10:50:58.855378 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:58.855003    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v8pkg" event={"ID":"a07d3f20-9317-46cb-aab7-097ac29c23a3","Type":"ContainerStarted","Data":"9b2efedaa86c5c457dbbda17c80dab4eb4321d5199ba4407db88c4c5a0ba593f"}
May 29 10:50:59.860389 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:59.860344    2547 generic.go:358] "Generic (PLEG): container finished" podID="c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7" containerID="3eda1d8e76b12f0d8903a29db0bd4ae4401aa83bdbf91b16e79e1af0e589e176" exitCode=0
May 29 10:50:59.860389 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:50:59.860398    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm79t" event={"ID":"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7","Type":"ContainerDied","Data":"3eda1d8e76b12f0d8903a29db0bd4ae4401aa83bdbf91b16e79e1af0e589e176"}
May 29 10:51:00.330962 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:00.330923    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:51:00.331155 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:00.330977    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:51:00.331155 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:00.331048    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:00.331155 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:00.331087    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 10:51:00.331155 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:00.331147    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls podName:8da26b43-5619-444b-929c-a54e16f29818 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:04.331128224 +0000 UTC m=+43.207690435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls") pod "dns-default-wzwmb" (UID: "8da26b43-5619-444b-929c-a54e16f29818") : secret "dns-default-metrics-tls" not found
May 29 10:51:00.331381 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:00.331164    2547 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 10:51:00.331381 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:00.331185    2547 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9cb56d556-8drnj: secret "image-registry-tls" not found
May 29 10:51:00.331381 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:00.331188    2547 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 10:51:00.331381 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:00.331247    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls podName:d6084d0f-a832-4de8-a7db-0489742556e1 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:04.331217495 +0000 UTC m=+43.207779701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls") pod "image-registry-9cb56d556-8drnj" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1") : secret "image-registry-tls" not found
May 29 10:51:00.331381 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:00.331267    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert podName:5c4e7979-7e81-4939-a895-b934805b1275 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:04.331256647 +0000 UTC m=+43.207818859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert") pod "ingress-canary-5dlbc" (UID: "5c4e7979-7e81-4939-a895-b934805b1275") : secret "canary-serving-cert" not found
May 29 10:51:00.865977 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:00.865934    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm79t" event={"ID":"c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7","Type":"ContainerStarted","Data":"1e07c7e3958d781c33d8ba700622bc2005f5e915fa34423385e23e0ccc9151de"}
May 29 10:51:00.890856 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:00.890805    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qm79t" podStartSLOduration=6.514360468 podStartE2EDuration="39.890791324s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:50:24.345606794 +0000 UTC m=+3.222169001" lastFinishedPulling="2026-05-29 10:50:57.722037638 +0000 UTC m=+36.598599857" observedRunningTime="2026-05-29 10:51:00.890019096 +0000 UTC m=+39.766581326" watchObservedRunningTime="2026-05-29 10:51:00.890791324 +0000 UTC m=+39.767353553"
May 29 10:51:01.869331 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:01.869285    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v8pkg" event={"ID":"a07d3f20-9317-46cb-aab7-097ac29c23a3","Type":"ContainerStarted","Data":"e5278385d9cc80d7b61f6695ac30f1761da966a06c2657b77a91a31e4aae00c0"}
May 29 10:51:01.885002 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:01.884955    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v8pkg" podStartSLOduration=33.309041197 podStartE2EDuration="36.884942198s" podCreationTimestamp="2026-05-29 10:50:25 +0000 UTC" firstStartedPulling="2026-05-29 10:50:58.069268984 +0000 UTC m=+36.945831190" lastFinishedPulling="2026-05-29 10:51:01.645169986 +0000 UTC m=+40.521732191" observedRunningTime="2026-05-29 10:51:01.884108502 +0000 UTC m=+40.760670727" watchObservedRunningTime="2026-05-29 10:51:01.884942198 +0000 UTC m=+40.761504425"
May 29 10:51:04.362951 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:04.362910    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:51:04.362951 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:04.362957    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:51:04.363389 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:04.363055    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:04.363389 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:04.363062    2547 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 10:51:04.363389 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:04.363131    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 10:51:04.363389 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:04.363135    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert podName:5c4e7979-7e81-4939-a895-b934805b1275 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:12.363115386 +0000 UTC m=+51.239677592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert") pod "ingress-canary-5dlbc" (UID: "5c4e7979-7e81-4939-a895-b934805b1275") : secret "canary-serving-cert" not found
May 29 10:51:04.363389 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:04.363180    2547 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 10:51:04.363389 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:04.363194    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls podName:8da26b43-5619-444b-929c-a54e16f29818 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:12.363178376 +0000 UTC m=+51.239740589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls") pod "dns-default-wzwmb" (UID: "8da26b43-5619-444b-929c-a54e16f29818") : secret "dns-default-metrics-tls" not found
May 29 10:51:04.363389 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:04.363193    2547 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9cb56d556-8drnj: secret "image-registry-tls" not found
May 29 10:51:04.363389 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:04.363225    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls podName:d6084d0f-a832-4de8-a7db-0489742556e1 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:12.363217162 +0000 UTC m=+51.239779372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls") pod "image-registry-9cb56d556-8drnj" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1") : secret "image-registry-tls" not found
May 29 10:51:12.428123 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:12.428080    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:51:12.428666 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:12.428136    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:51:12.428666 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:12.428164    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:12.428666 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:12.428257    2547 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 10:51:12.428666 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:12.428323    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert podName:5c4e7979-7e81-4939-a895-b934805b1275 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:28.428303623 +0000 UTC m=+67.304865852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert") pod "ingress-canary-5dlbc" (UID: "5c4e7979-7e81-4939-a895-b934805b1275") : secret "canary-serving-cert" not found
May 29 10:51:12.428666 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:12.428330    2547 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 10:51:12.428666 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:12.428342    2547 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9cb56d556-8drnj: secret "image-registry-tls" not found
May 29 10:51:12.428666 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:12.428358    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 10:51:12.428666 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:12.428376    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls podName:d6084d0f-a832-4de8-a7db-0489742556e1 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:28.428364777 +0000 UTC m=+67.304926998 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls") pod "image-registry-9cb56d556-8drnj" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1") : secret "image-registry-tls" not found
May 29 10:51:12.428666 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:12.428426    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls podName:8da26b43-5619-444b-929c-a54e16f29818 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:28.428407395 +0000 UTC m=+67.304969611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls") pod "dns-default-wzwmb" (UID: "8da26b43-5619-444b-929c-a54e16f29818") : secret "dns-default-metrics-tls" not found
May 29 10:51:13.844100 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:13.844070    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ghsp2_ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041/dns-node-resolver/0.log"
May 29 10:51:14.647593 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:14.647560    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2n89j_e8e00f00-647c-463b-b317-aa670bde2fd4/node-ca/0.log"
May 29 10:51:20.847157 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:20.847121    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c8682"
May 29 10:51:27.545550 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.545507    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:51:27.547954 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.547937    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 29 10:51:27.559577 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.559551    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7883155e-3d26-41dc-8c39-a954501459b5-metrics-certs\") pod \"network-metrics-daemon-fqncf\" (UID: \"7883155e-3d26-41dc-8c39-a954501459b5\") " pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:51:27.641399 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.641367    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cqxtm\""
May 29 10:51:27.646393 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.646375    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:51:27.648775 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.648761    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 29 10:51:27.649784 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.649768    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqncf"
May 29 10:51:27.659856 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.659835    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 29 10:51:27.670039 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.670016    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64gfw\" (UniqueName: \"kubernetes.io/projected/532d0df8-fdbb-40c5-8f95-d6f2975fc47e-kube-api-access-64gfw\") pod \"network-check-target-4524p\" (UID: \"532d0df8-fdbb-40c5-8f95-d6f2975fc47e\") " pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:51:27.765882 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.765853    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fqncf"]
May 29 10:51:27.768863 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:51:27.768833    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7883155e_3d26_41dc_8c39_a954501459b5.slice/crio-c94977a4dddf54a493f616552385d8bae1f8adccb89edce052265079acd539b3 WatchSource:0}: Error finding container c94977a4dddf54a493f616552385d8bae1f8adccb89edce052265079acd539b3: Status 404 returned error can't find the container with id c94977a4dddf54a493f616552385d8bae1f8adccb89edce052265079acd539b3
May 29 10:51:27.918234 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.918151    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqncf" event={"ID":"7883155e-3d26-41dc-8c39-a954501459b5","Type":"ContainerStarted","Data":"c94977a4dddf54a493f616552385d8bae1f8adccb89edce052265079acd539b3"}
May 29 10:51:27.934574 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.934547    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d5k7t\""
May 29 10:51:27.942550 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:27.942530    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:51:28.059595 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.059565    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4524p"]
May 29 10:51:28.063413 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:51:28.063380    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532d0df8_fdbb_40c5_8f95_d6f2975fc47e.slice/crio-ab29f5c242ba6fc80b770d3489462bf1e53d4f4ef55b62ca58a074f43549d268 WatchSource:0}: Error finding container ab29f5c242ba6fc80b770d3489462bf1e53d4f4ef55b62ca58a074f43549d268: Status 404 returned error can't find the container with id ab29f5c242ba6fc80b770d3489462bf1e53d4f4ef55b62ca58a074f43549d268
May 29 10:51:28.455174 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.455137    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:51:28.455414 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.455183    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:51:28.455414 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.455213    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:28.457463 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.457434    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8da26b43-5619-444b-929c-a54e16f29818-metrics-tls\") pod \"dns-default-wzwmb\" (UID: \"8da26b43-5619-444b-929c-a54e16f29818\") " pod="openshift-dns/dns-default-wzwmb"
May 29 10:51:28.457829 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.457809    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c4e7979-7e81-4939-a895-b934805b1275-cert\") pod \"ingress-canary-5dlbc\" (UID: \"5c4e7979-7e81-4939-a895-b934805b1275\") " pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:51:28.457829 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.457821    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"image-registry-9cb56d556-8drnj\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") " pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:28.665133 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.665103    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5b8kn\""
May 29 10:51:28.673058 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.673024    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:28.683810 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.683783    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fk5jh\""
May 29 10:51:28.692748 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.692728    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5dlbc"
May 29 10:51:28.699761 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.699739    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4jnj\""
May 29 10:51:28.707939 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.707796    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wzwmb"
May 29 10:51:28.933839 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:28.932045    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4524p" event={"ID":"532d0df8-fdbb-40c5-8f95-d6f2975fc47e","Type":"ContainerStarted","Data":"ab29f5c242ba6fc80b770d3489462bf1e53d4f4ef55b62ca58a074f43549d268"}
May 29 10:51:29.019232 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.018812    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wzwmb"]
May 29 10:51:29.024588 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:51:29.024561    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da26b43_5619_444b_929c_a54e16f29818.slice/crio-0b13a01ebe356797c16680c5ea2382987014987688f886df86bf4bcc349594aa WatchSource:0}: Error finding container 0b13a01ebe356797c16680c5ea2382987014987688f886df86bf4bcc349594aa: Status 404 returned error can't find the container with id 0b13a01ebe356797c16680c5ea2382987014987688f886df86bf4bcc349594aa
May 29 10:51:29.252357 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.252269    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5dlbc"]
May 29 10:51:29.256265 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.256224    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9cb56d556-8drnj"]
May 29 10:51:29.258913 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:51:29.258877    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4e7979_7e81_4939_a895_b934805b1275.slice/crio-72c93f588d466eae0edf5e39057bb1f40822f5c3d32b3932e1b4e58de4337287 WatchSource:0}: Error finding container 72c93f588d466eae0edf5e39057bb1f40822f5c3d32b3932e1b4e58de4337287: Status 404 returned error can't find the container with id 72c93f588d466eae0edf5e39057bb1f40822f5c3d32b3932e1b4e58de4337287
May 29 10:51:29.262211 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:51:29.262182    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6084d0f_a832_4de8_a7db_0489742556e1.slice/crio-ad5ffbf686821a4d3fc62ff17dd55f9071eb44ace3b206dc91cb873efd44cd0b WatchSource:0}: Error finding container ad5ffbf686821a4d3fc62ff17dd55f9071eb44ace3b206dc91cb873efd44cd0b: Status 404 returned error can't find the container with id ad5ffbf686821a4d3fc62ff17dd55f9071eb44ace3b206dc91cb873efd44cd0b
May 29 10:51:29.936013 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.935953    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5dlbc" event={"ID":"5c4e7979-7e81-4939-a895-b934805b1275","Type":"ContainerStarted","Data":"72c93f588d466eae0edf5e39057bb1f40822f5c3d32b3932e1b4e58de4337287"}
May 29 10:51:29.939713 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.939460    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqncf" event={"ID":"7883155e-3d26-41dc-8c39-a954501459b5","Type":"ContainerStarted","Data":"be064df96739063284dc8b950aed9d7004b9805a7f1bd30aee0d854df912f682"}
May 29 10:51:29.939713 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.939551    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqncf" event={"ID":"7883155e-3d26-41dc-8c39-a954501459b5","Type":"ContainerStarted","Data":"dea15d1490c18996128775167272e5bf1b6d45c7baf1a1916d20fd644b878414"}
May 29 10:51:29.941807 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.941777    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wzwmb" event={"ID":"8da26b43-5619-444b-929c-a54e16f29818","Type":"ContainerStarted","Data":"0b13a01ebe356797c16680c5ea2382987014987688f886df86bf4bcc349594aa"}
May 29 10:51:29.943372 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.943339    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9cb56d556-8drnj" event={"ID":"d6084d0f-a832-4de8-a7db-0489742556e1","Type":"ContainerStarted","Data":"76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e"}
May 29 10:51:29.943484 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.943374    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9cb56d556-8drnj" event={"ID":"d6084d0f-a832-4de8-a7db-0489742556e1","Type":"ContainerStarted","Data":"ad5ffbf686821a4d3fc62ff17dd55f9071eb44ace3b206dc91cb873efd44cd0b"}
May 29 10:51:29.944079 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.944040    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:29.956815 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.956762    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fqncf" podStartSLOduration=67.85644012 podStartE2EDuration="1m8.956745725s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:51:27.770699309 +0000 UTC m=+66.647261515" lastFinishedPulling="2026-05-29 10:51:28.871004899 +0000 UTC m=+67.747567120" observedRunningTime="2026-05-29 10:51:29.955172364 +0000 UTC m=+68.831734595" watchObservedRunningTime="2026-05-29 10:51:29.956745725 +0000 UTC m=+68.833307953"
May 29 10:51:29.973925 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:29.973870    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9cb56d556-8drnj" podStartSLOduration=38.973852996 podStartE2EDuration="38.973852996s" podCreationTimestamp="2026-05-29 10:50:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:51:29.972665503 +0000 UTC m=+68.849227732" watchObservedRunningTime="2026-05-29 10:51:29.973852996 +0000 UTC m=+68.850415226"
May 29 10:51:31.790633 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.790599    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wlpw2"]
May 29 10:51:31.794859 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.794834    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.798120 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.798094    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zmxw7\""
May 29 10:51:31.798269 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.798127    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 29 10:51:31.798413 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.798395    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 29 10:51:31.798513 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.798490    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 29 10:51:31.798628 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.798595    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 29 10:51:31.805904 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.805880    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wlpw2"]
May 29 10:51:31.875885 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.875856    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrs8\" (UniqueName: \"kubernetes.io/projected/afddd537-f5d6-401f-95d7-5ad4c38ff18b-kube-api-access-ktrs8\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.876069 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.875899    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afddd537-f5d6-401f-95d7-5ad4c38ff18b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.876069 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.875989    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afddd537-f5d6-401f-95d7-5ad4c38ff18b-data-volume\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.876174 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.876075    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afddd537-f5d6-401f-95d7-5ad4c38ff18b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.876174 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.876113    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afddd537-f5d6-401f-95d7-5ad4c38ff18b-crio-socket\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.897349 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.897313    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9cb56d556-8drnj"]
May 29 10:51:31.904657 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.904619    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-c7hjh"]
May 29 10:51:31.907824 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.907798    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-c7hjh"
May 29 10:51:31.911933 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.911907    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wq8tn\""
May 29 10:51:31.912562 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.912538    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 29 10:51:31.912684 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.912565    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 29 10:51:31.933713 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.933686    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-c7hjh"]
May 29 10:51:31.977105 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.977083    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95wg\" (UniqueName: \"kubernetes.io/projected/6b150d8b-02db-466e-bd9f-8d875e33a0e2-kube-api-access-z95wg\") pod \"downloads-57bd96b888-c7hjh\" (UID: \"6b150d8b-02db-466e-bd9f-8d875e33a0e2\") " pod="openshift-console/downloads-57bd96b888-c7hjh"
May 29 10:51:31.977219 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.977132    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrs8\" (UniqueName: \"kubernetes.io/projected/afddd537-f5d6-401f-95d7-5ad4c38ff18b-kube-api-access-ktrs8\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.977219 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.977200    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afddd537-f5d6-401f-95d7-5ad4c38ff18b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.977345 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.977288    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afddd537-f5d6-401f-95d7-5ad4c38ff18b-data-volume\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.977345 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.977338    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afddd537-f5d6-401f-95d7-5ad4c38ff18b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.977443 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.977374    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afddd537-f5d6-401f-95d7-5ad4c38ff18b-crio-socket\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.977500 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.977481    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afddd537-f5d6-401f-95d7-5ad4c38ff18b-crio-socket\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.977601 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.977580    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afddd537-f5d6-401f-95d7-5ad4c38ff18b-data-volume\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.977782 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.977765    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afddd537-f5d6-401f-95d7-5ad4c38ff18b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.979511 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.979485    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afddd537-f5d6-401f-95d7-5ad4c38ff18b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:31.994987 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:31.994959    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrs8\" (UniqueName: \"kubernetes.io/projected/afddd537-f5d6-401f-95d7-5ad4c38ff18b-kube-api-access-ktrs8\") pod \"insights-runtime-extractor-wlpw2\" (UID: \"afddd537-f5d6-401f-95d7-5ad4c38ff18b\") " pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:32.078640 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.078596    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z95wg\" (UniqueName: \"kubernetes.io/projected/6b150d8b-02db-466e-bd9f-8d875e33a0e2-kube-api-access-z95wg\") pod \"downloads-57bd96b888-c7hjh\" (UID: \"6b150d8b-02db-466e-bd9f-8d875e33a0e2\") " pod="openshift-console/downloads-57bd96b888-c7hjh"
May 29 10:51:32.088633 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.088590    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95wg\" (UniqueName: \"kubernetes.io/projected/6b150d8b-02db-466e-bd9f-8d875e33a0e2-kube-api-access-z95wg\") pod \"downloads-57bd96b888-c7hjh\" (UID: \"6b150d8b-02db-466e-bd9f-8d875e33a0e2\") " pod="openshift-console/downloads-57bd96b888-c7hjh"
May 29 10:51:32.105886 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.105851    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wlpw2"
May 29 10:51:32.219708 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.219562    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-c7hjh"
May 29 10:51:32.306990 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.306679    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wlpw2"]
May 29 10:51:32.309456 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:51:32.309414    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafddd537_f5d6_401f_95d7_5ad4c38ff18b.slice/crio-db88c00fbac6d83c46d1bb0943526d6598e9a1df71c5b956614c11ef5a904070 WatchSource:0}: Error finding container db88c00fbac6d83c46d1bb0943526d6598e9a1df71c5b956614c11ef5a904070: Status 404 returned error can't find the container with id db88c00fbac6d83c46d1bb0943526d6598e9a1df71c5b956614c11ef5a904070
May 29 10:51:32.356424 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.356373    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-c7hjh"]
May 29 10:51:32.359988 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:51:32.359961    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b150d8b_02db_466e_bd9f_8d875e33a0e2.slice/crio-ff169e17e78858dd15526248e7b220e025c45b2109aeb19ab8db2293d2e7e1e4 WatchSource:0}: Error finding container ff169e17e78858dd15526248e7b220e025c45b2109aeb19ab8db2293d2e7e1e4: Status 404 returned error can't find the container with id ff169e17e78858dd15526248e7b220e025c45b2109aeb19ab8db2293d2e7e1e4
May 29 10:51:32.952745 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.952702    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wzwmb" event={"ID":"8da26b43-5619-444b-929c-a54e16f29818","Type":"ContainerStarted","Data":"2d031283ef2fc5d3fc216f2807fb6017b0d2a3efb7d0a1bd6d72e0e8a585e78a"}
May 29 10:51:32.953307 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.952752    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wzwmb" event={"ID":"8da26b43-5619-444b-929c-a54e16f29818","Type":"ContainerStarted","Data":"0c25659ec96c58c6faf829eaf98a7d8a34ee34db3c88baed31edc92876c290f1"}
May 29 10:51:32.953307 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.952880    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wzwmb"
May 29 10:51:32.954377 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.954344    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4524p" event={"ID":"532d0df8-fdbb-40c5-8f95-d6f2975fc47e","Type":"ContainerStarted","Data":"ad390a936805000cd704253290d9fdb23933acedf5e7d69a9ad4e72df3cfe72b"}
May 29 10:51:32.954505 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.954481    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:51:32.955544 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.955512    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-c7hjh" event={"ID":"6b150d8b-02db-466e-bd9f-8d875e33a0e2","Type":"ContainerStarted","Data":"ff169e17e78858dd15526248e7b220e025c45b2109aeb19ab8db2293d2e7e1e4"}
May 29 10:51:32.956924 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.956898    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5dlbc" event={"ID":"5c4e7979-7e81-4939-a895-b934805b1275","Type":"ContainerStarted","Data":"1f686c65fba938c6a8108c21a0be26bafbe561e262d537b87cad188bfc73d1b7"}
May 29 10:51:32.958169 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.958145    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wlpw2" event={"ID":"afddd537-f5d6-401f-95d7-5ad4c38ff18b","Type":"ContainerStarted","Data":"8874aed6d492c8c1bede9d7029ada4e01ea25b63e3c3d4c095b93cd584912e0f"}
May 29 10:51:32.958289 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.958175    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wlpw2" event={"ID":"afddd537-f5d6-401f-95d7-5ad4c38ff18b","Type":"ContainerStarted","Data":"db88c00fbac6d83c46d1bb0943526d6598e9a1df71c5b956614c11ef5a904070"}
May 29 10:51:32.970173 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:32.970121    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-wzwmb" podStartSLOduration=34.005050756 podStartE2EDuration="36.970104498s" podCreationTimestamp="2026-05-29 10:50:56 +0000 UTC" firstStartedPulling="2026-05-29 10:51:29.027016811 +0000 UTC m=+67.903579021" lastFinishedPulling="2026-05-29 10:51:31.992070547 +0000 UTC m=+70.868632763" observedRunningTime="2026-05-29 10:51:32.968197075 +0000 UTC m=+71.844759302" watchObservedRunningTime="2026-05-29 10:51:32.970104498 +0000 UTC m=+71.846666727"
May 29 10:51:33.000304 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:33.000203    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4524p" podStartSLOduration=68.074390861 podStartE2EDuration="1m12.000184329s" podCreationTimestamp="2026-05-29 10:50:21 +0000 UTC" firstStartedPulling="2026-05-29 10:51:28.065199991 +0000 UTC m=+66.941762200" lastFinishedPulling="2026-05-29 10:51:31.990993461 +0000 UTC m=+70.867555668" observedRunningTime="2026-05-29 10:51:32.985769551 +0000 UTC m=+71.862331779" watchObservedRunningTime="2026-05-29 10:51:33.000184329 +0000 UTC m=+71.876746576"
May 29 10:51:33.001088 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:33.001052    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5dlbc" podStartSLOduration=34.266783718 podStartE2EDuration="37.001044165s" podCreationTimestamp="2026-05-29 10:50:56 +0000 UTC" firstStartedPulling="2026-05-29 10:51:29.261316239 +0000 UTC m=+68.137878466" lastFinishedPulling="2026-05-29 10:51:31.995576708 +0000 UTC m=+70.872138913" observedRunningTime="2026-05-29 10:51:32.999832356 +0000 UTC m=+71.876394598" watchObservedRunningTime="2026-05-29 10:51:33.001044165 +0000 UTC m=+71.877606391"
May 29 10:51:33.964889 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:33.964747    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wlpw2" event={"ID":"afddd537-f5d6-401f-95d7-5ad4c38ff18b","Type":"ContainerStarted","Data":"1d7002ba81a864ba761187bc182284e5673d3ebbc783211bc5ef1e31cdb71292"}
May 29 10:51:34.969259 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:34.969160    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wlpw2" event={"ID":"afddd537-f5d6-401f-95d7-5ad4c38ff18b","Type":"ContainerStarted","Data":"126b84f96fef7dc6ce5b3bd096fc52d8313a11318f45c0ebf37985ec35b95881"}
May 29 10:51:34.986841 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:34.986782    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wlpw2" podStartSLOduration=1.746796925 podStartE2EDuration="3.986766924s" podCreationTimestamp="2026-05-29 10:51:31 +0000 UTC" firstStartedPulling="2026-05-29 10:51:32.365106061 +0000 UTC m=+71.241668266" lastFinishedPulling="2026-05-29 10:51:34.605076059 +0000 UTC m=+73.481638265" observedRunningTime="2026-05-29 10:51:34.985206647 +0000 UTC m=+73.861768877" watchObservedRunningTime="2026-05-29 10:51:34.986766924 +0000 UTC m=+73.863329152"
May 29 10:51:38.968840 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.968808    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-56f656d898-b7qlm"]
May 29 10:51:38.973230 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.973204    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:38.976525 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.976502    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 29 10:51:38.976525 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.976515    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 29 10:51:38.976715 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.976628    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zln56\""
May 29 10:51:38.976863 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.976839    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 29 10:51:38.976863 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.976870    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 29 10:51:38.977024 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.976873    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 29 10:51:38.986582 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.986558    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 29 10:51:38.986799 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:38.986698    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f656d898-b7qlm"]
May 29 10:51:39.033915 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.033879    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-service-ca\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.033915 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.033922    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-serving-cert\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.034164 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.033960    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-oauth-config\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.034164 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.033997    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tk8\" (UniqueName: \"kubernetes.io/projected/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-kube-api-access-65tk8\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.034164 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.034023    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-oauth-serving-cert\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.034164 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.034060    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-config\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.034164 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.034108    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-trusted-ca-bundle\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.135495 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.135458    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-oauth-config\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.135724 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.135511    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-65tk8\" (UniqueName: \"kubernetes.io/projected/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-kube-api-access-65tk8\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.135724 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.135544    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-oauth-serving-cert\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.135724 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.135578    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-config\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.135724 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.135599    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-trusted-ca-bundle\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.135724 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.135635    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-service-ca\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.135724 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.135661    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-serving-cert\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.136382 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.136350    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-oauth-serving-cert\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.136499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.136383    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-config\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.136553 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.136497    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-service-ca\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.136795 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.136772    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-trusted-ca-bundle\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.138233 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.138212    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-oauth-config\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.138233 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.138223    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-serving-cert\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.143590 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.143569    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tk8\" (UniqueName: \"kubernetes.io/projected/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-kube-api-access-65tk8\") pod \"console-56f656d898-b7qlm\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") " pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.292972 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.292868    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:39.436702 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.436523    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f656d898-b7qlm"]
May 29 10:51:39.439813 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:51:39.439783    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6bb824_3dc1_4020_a75d_d5a78fa07ec1.slice/crio-8593748ecd6c90e160b460bc46766027df81a46ccd2cda875b1bac07c1d44842 WatchSource:0}: Error finding container 8593748ecd6c90e160b460bc46766027df81a46ccd2cda875b1bac07c1d44842: Status 404 returned error can't find the container with id 8593748ecd6c90e160b460bc46766027df81a46ccd2cda875b1bac07c1d44842
May 29 10:51:39.991100 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:39.991057    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f656d898-b7qlm" event={"ID":"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1","Type":"ContainerStarted","Data":"8593748ecd6c90e160b460bc46766027df81a46ccd2cda875b1bac07c1d44842"}
May 29 10:51:42.967963 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:42.967928    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wzwmb"
May 29 10:51:43.001227 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:43.001191    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f656d898-b7qlm" event={"ID":"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1","Type":"ContainerStarted","Data":"5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4"}
May 29 10:51:43.018275 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:43.018204    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-56f656d898-b7qlm" podStartSLOduration=2.173823991 podStartE2EDuration="5.018186296s" podCreationTimestamp="2026-05-29 10:51:38 +0000 UTC" firstStartedPulling="2026-05-29 10:51:39.44210177 +0000 UTC m=+78.318663977" lastFinishedPulling="2026-05-29 10:51:42.286464073 +0000 UTC m=+81.163026282" observedRunningTime="2026-05-29 10:51:43.017363625 +0000 UTC m=+81.893925853" watchObservedRunningTime="2026-05-29 10:51:43.018186296 +0000 UTC m=+81.894748526"
May 29 10:51:49.293407 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:49.293362    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:49.293407 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:49.293415    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:49.298976 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:49.298942    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:50.025518 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:50.025477    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-c7hjh" event={"ID":"6b150d8b-02db-466e-bd9f-8d875e33a0e2","Type":"ContainerStarted","Data":"5ea097b3ae5fb416a95ab5df1923f50a502611be9c4c3c0de923a55a47931082"}
May 29 10:51:50.026020 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:50.025974    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-c7hjh"
May 29 10:51:50.030180 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:50.030153    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:51:50.036792 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:50.036772    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-c7hjh"
May 29 10:51:50.052022 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:50.051972    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-c7hjh" podStartSLOduration=1.812662242 podStartE2EDuration="19.051954871s" podCreationTimestamp="2026-05-29 10:51:31 +0000 UTC" firstStartedPulling="2026-05-29 10:51:32.361871186 +0000 UTC m=+71.238433392" lastFinishedPulling="2026-05-29 10:51:49.6011638 +0000 UTC m=+88.477726021" observedRunningTime="2026-05-29 10:51:50.05091924 +0000 UTC m=+88.927481471" watchObservedRunningTime="2026-05-29 10:51:50.051954871 +0000 UTC m=+88.928517100"
May 29 10:51:52.965102 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:52.965071    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:57.979350 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:57.979303    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9cb56d556-8drnj" podUID="d6084d0f-a832-4de8-a7db-0489742556e1" containerName="registry" containerID="cri-o://76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e" gracePeriod=30
May 29 10:51:58.104166 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.104128    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"]
May 29 10:51:58.127754 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.127699    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"]
May 29 10:51:58.152973 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.152762    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"
May 29 10:51:58.155383 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.155350    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x8l5j\""
May 29 10:51:58.155727 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.155702    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 29 10:51:58.256526 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.256501    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:58.293884 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.293849    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1621fbfd-dd5c-42c9-b036-4410ffad4608-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8\" (UID: \"1621fbfd-dd5c-42c9-b036-4410ffad4608\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"
May 29 10:51:58.394327 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394288    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") pod \"d6084d0f-a832-4de8-a7db-0489742556e1\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") "
May 29 10:51:58.394327 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394329    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-image-registry-private-configuration\") pod \"d6084d0f-a832-4de8-a7db-0489742556e1\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") "
May 29 10:51:58.394598 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394365    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6084d0f-a832-4de8-a7db-0489742556e1-ca-trust-extracted\") pod \"d6084d0f-a832-4de8-a7db-0489742556e1\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") "
May 29 10:51:58.394598 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394384    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-bound-sa-token\") pod \"d6084d0f-a832-4de8-a7db-0489742556e1\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") "
May 29 10:51:58.394598 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394417    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-trusted-ca\") pod \"d6084d0f-a832-4de8-a7db-0489742556e1\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") "
May 29 10:51:58.394598 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394461    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnpm7\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-kube-api-access-qnpm7\") pod \"d6084d0f-a832-4de8-a7db-0489742556e1\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") "
May 29 10:51:58.394598 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394490    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-installation-pull-secrets\") pod \"d6084d0f-a832-4de8-a7db-0489742556e1\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") "
May 29 10:51:58.394598 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394530    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-registry-certificates\") pod \"d6084d0f-a832-4de8-a7db-0489742556e1\" (UID: \"d6084d0f-a832-4de8-a7db-0489742556e1\") "
May 29 10:51:58.394915 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394695    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1621fbfd-dd5c-42c9-b036-4410ffad4608-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8\" (UID: \"1621fbfd-dd5c-42c9-b036-4410ffad4608\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"
May 29 10:51:58.394915 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:58.394821    2547 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 29 10:51:58.394915 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.394842    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d6084d0f-a832-4de8-a7db-0489742556e1" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:51:58.394915 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:58.394888    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1621fbfd-dd5c-42c9-b036-4410ffad4608-tls-certificates podName:1621fbfd-dd5c-42c9-b036-4410ffad4608 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:58.894866817 +0000 UTC m=+97.771429024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1621fbfd-dd5c-42c9-b036-4410ffad4608-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8" (UID: "1621fbfd-dd5c-42c9-b036-4410ffad4608") : secret "prometheus-operator-admission-webhook-tls" not found
May 29 10:51:58.395501 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.395114    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d6084d0f-a832-4de8-a7db-0489742556e1" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:51:58.397323 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.397298    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d6084d0f-a832-4de8-a7db-0489742556e1" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:51:58.397649 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.397601    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d6084d0f-a832-4de8-a7db-0489742556e1" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:51:58.397744 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.397719    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-kube-api-access-qnpm7" (OuterVolumeSpecName: "kube-api-access-qnpm7") pod "d6084d0f-a832-4de8-a7db-0489742556e1" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1"). InnerVolumeSpecName "kube-api-access-qnpm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:51:58.398128 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.398093    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d6084d0f-a832-4de8-a7db-0489742556e1" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:51:58.398842 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.398814    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d6084d0f-a832-4de8-a7db-0489742556e1" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:51:58.404124 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.404098    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6084d0f-a832-4de8-a7db-0489742556e1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d6084d0f-a832-4de8-a7db-0489742556e1" (UID: "d6084d0f-a832-4de8-a7db-0489742556e1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:51:58.495507 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.495428    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:51:58.495507 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.495456    2547 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-image-registry-private-configuration\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:51:58.495507 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.495467    2547 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6084d0f-a832-4de8-a7db-0489742556e1-ca-trust-extracted\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:51:58.495507 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.495477    2547 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-bound-sa-token\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:51:58.495507 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.495486    2547 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-trusted-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:51:58.495507 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.495494    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnpm7\" (UniqueName: \"kubernetes.io/projected/d6084d0f-a832-4de8-a7db-0489742556e1-kube-api-access-qnpm7\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:51:58.495507 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.495503    2547 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6084d0f-a832-4de8-a7db-0489742556e1-installation-pull-secrets\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:51:58.495507 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.495511    2547 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6084d0f-a832-4de8-a7db-0489742556e1-registry-certificates\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:51:58.898348 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:58.898313    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1621fbfd-dd5c-42c9-b036-4410ffad4608-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8\" (UID: \"1621fbfd-dd5c-42c9-b036-4410ffad4608\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"
May 29 10:51:58.898554 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:58.898462    2547 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 29 10:51:58.898554 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:58.898523    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1621fbfd-dd5c-42c9-b036-4410ffad4608-tls-certificates podName:1621fbfd-dd5c-42c9-b036-4410ffad4608 nodeName:}" failed. No retries permitted until 2026-05-29 10:51:59.898504158 +0000 UTC m=+98.775066364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1621fbfd-dd5c-42c9-b036-4410ffad4608-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8" (UID: "1621fbfd-dd5c-42c9-b036-4410ffad4608") : secret "prometheus-operator-admission-webhook-tls" not found
May 29 10:51:59.054323 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.054287    2547 generic.go:358] "Generic (PLEG): container finished" podID="d6084d0f-a832-4de8-a7db-0489742556e1" containerID="76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e" exitCode=0
May 29 10:51:59.054882 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.054345    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9cb56d556-8drnj" event={"ID":"d6084d0f-a832-4de8-a7db-0489742556e1","Type":"ContainerDied","Data":"76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e"}
May 29 10:51:59.054882 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.054380    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9cb56d556-8drnj" event={"ID":"d6084d0f-a832-4de8-a7db-0489742556e1","Type":"ContainerDied","Data":"ad5ffbf686821a4d3fc62ff17dd55f9071eb44ace3b206dc91cb873efd44cd0b"}
May 29 10:51:59.054882 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.054399    2547 scope.go:117] "RemoveContainer" containerID="76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e"
May 29 10:51:59.054882 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.054350    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9cb56d556-8drnj"
May 29 10:51:59.062951 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.062745    2547 scope.go:117] "RemoveContainer" containerID="76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e"
May 29 10:51:59.063508 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:51:59.063445    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e\": container with ID starting with 76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e not found: ID does not exist" containerID="76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e"
May 29 10:51:59.063786 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.063491    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e"} err="failed to get container status \"76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e\": rpc error: code = NotFound desc = could not find container \"76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e\": container with ID starting with 76856ca20bf43c8031f085a5eab8328a6b4206330b56d6d279296b54e6be879e not found: ID does not exist"
May 29 10:51:59.088488 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.088467    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9cb56d556-8drnj"]
May 29 10:51:59.097709 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.097690    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9cb56d556-8drnj"]
May 29 10:51:59.715868 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.715830    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6084d0f-a832-4de8-a7db-0489742556e1" path="/var/lib/kubelet/pods/d6084d0f-a832-4de8-a7db-0489742556e1/volumes"
May 29 10:51:59.904676 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.904641    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1621fbfd-dd5c-42c9-b036-4410ffad4608-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8\" (UID: \"1621fbfd-dd5c-42c9-b036-4410ffad4608\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"
May 29 10:51:59.907036 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.907005    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1621fbfd-dd5c-42c9-b036-4410ffad4608-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8\" (UID: \"1621fbfd-dd5c-42c9-b036-4410ffad4608\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"
May 29 10:51:59.962776 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:51:59.962751    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"
May 29 10:52:00.083868 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:00.083839    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"]
May 29 10:52:00.087068 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:52:00.087041    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1621fbfd_dd5c_42c9_b036_4410ffad4608.slice/crio-62a44f601dcbb5819b6c2701395bc99d249b9a5934d03a6e889d079ec0f7cd0e WatchSource:0}: Error finding container 62a44f601dcbb5819b6c2701395bc99d249b9a5934d03a6e889d079ec0f7cd0e: Status 404 returned error can't find the container with id 62a44f601dcbb5819b6c2701395bc99d249b9a5934d03a6e889d079ec0f7cd0e
May 29 10:52:01.061700 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:01.061667    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8" event={"ID":"1621fbfd-dd5c-42c9-b036-4410ffad4608","Type":"ContainerStarted","Data":"62a44f601dcbb5819b6c2701395bc99d249b9a5934d03a6e889d079ec0f7cd0e"}
May 29 10:52:02.065585 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:02.065548    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8" event={"ID":"1621fbfd-dd5c-42c9-b036-4410ffad4608","Type":"ContainerStarted","Data":"715bd3ec43faf26e492e784fc3e8cc1f3c194ed89bdb05e1f28ed095a0f62276"}
May 29 10:52:02.066067 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:02.065785    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"
May 29 10:52:02.070448 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:02.070422    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8"
May 29 10:52:02.081781 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:02.081736    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8" podStartSLOduration=2.639176488 podStartE2EDuration="4.081723193s" podCreationTimestamp="2026-05-29 10:51:58 +0000 UTC" firstStartedPulling="2026-05-29 10:52:00.088793932 +0000 UTC m=+98.965356142" lastFinishedPulling="2026-05-29 10:52:01.531340629 +0000 UTC m=+100.407902847" observedRunningTime="2026-05-29 10:52:02.080657487 +0000 UTC m=+100.957219718" watchObservedRunningTime="2026-05-29 10:52:02.081723193 +0000 UTC m=+100.958285420"
May 29 10:52:03.155221 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.155189    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"]
May 29 10:52:03.155617 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.155416    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6084d0f-a832-4de8-a7db-0489742556e1" containerName="registry"
May 29 10:52:03.155617 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.155428    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6084d0f-a832-4de8-a7db-0489742556e1" containerName="registry"
May 29 10:52:03.155617 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.155504    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="d6084d0f-a832-4de8-a7db-0489742556e1" containerName="registry"
May 29 10:52:03.176344 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.176310    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"]
May 29 10:52:03.176485 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.176451    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.182311 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.182287    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 29 10:52:03.182311 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.182307    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 29 10:52:03.182506 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.182323    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 29 10:52:03.182769 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.182741    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fpbdx\""
May 29 10:52:03.182897 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.182881    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 29 10:52:03.183085 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.182984    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 29 10:52:03.330434 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.330398    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.330620 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.330451    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.330620 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.330524    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h72v\" (UniqueName: \"kubernetes.io/projected/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-kube-api-access-8h72v\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.330620 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.330591    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.431088 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.431007    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.431088 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.431044    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.431088 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.431088    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.431333 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.431110    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8h72v\" (UniqueName: \"kubernetes.io/projected/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-kube-api-access-8h72v\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.431830 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.431808    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.433445 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.433412    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.433567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.433547    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.438356 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.438334    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h72v\" (UniqueName: \"kubernetes.io/projected/82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb-kube-api-access-8h72v\") pod \"prometheus-operator-7779bbdd78-mcl5k\" (UID: \"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.489159 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.489128    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"
May 29 10:52:03.612815 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.612783    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k"]
May 29 10:52:03.615601 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:52:03.615573    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d2b2e9_6ba4_4cfb_b7b5_4ccf0d94cbfb.slice/crio-38e33b8558e291dc62120366168b209bd46e4d15ba52aa9714385708dd8b3c99 WatchSource:0}: Error finding container 38e33b8558e291dc62120366168b209bd46e4d15ba52aa9714385708dd8b3c99: Status 404 returned error can't find the container with id 38e33b8558e291dc62120366168b209bd46e4d15ba52aa9714385708dd8b3c99
May 29 10:52:03.967314 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:03.967287    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4524p"
May 29 10:52:04.072294 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:04.072260    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k" event={"ID":"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb","Type":"ContainerStarted","Data":"38e33b8558e291dc62120366168b209bd46e4d15ba52aa9714385708dd8b3c99"}
May 29 10:52:06.079524 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:06.079490    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k" event={"ID":"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb","Type":"ContainerStarted","Data":"acba87d559cbd72ed76f3e79f5250db5ed242fe697dd9408550e9ab67bfdf121"}
May 29 10:52:06.079524 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:06.079524    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k" event={"ID":"82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb","Type":"ContainerStarted","Data":"1f9737aa2b1be75be9a0b1df050603c4a401ed26ad6290eb060183f8bf4b936a"}
May 29 10:52:06.095255 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:06.095187    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mcl5k" podStartSLOduration=1.297883512 podStartE2EDuration="3.09517174s" podCreationTimestamp="2026-05-29 10:52:03 +0000 UTC" firstStartedPulling="2026-05-29 10:52:03.617626192 +0000 UTC m=+102.494188398" lastFinishedPulling="2026-05-29 10:52:05.41491442 +0000 UTC m=+104.291476626" observedRunningTime="2026-05-29 10:52:06.094391264 +0000 UTC m=+104.970953492" watchObservedRunningTime="2026-05-29 10:52:06.09517174 +0000 UTC m=+104.971733982"
May 29 10:52:07.503992 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.503958    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"]
May 29 10:52:07.532388 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.532362    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"]
May 29 10:52:07.532388 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.532388    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"]
May 29 10:52:07.532632 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.532546    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.535574 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.535553    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 29 10:52:07.535574 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.535571    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 29 10:52:07.535874 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.535859    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6kjb9\""
May 29 10:52:07.556376 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.556346    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"]
May 29 10:52:07.556555 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.556388    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-llk2k"]
May 29 10:52:07.556623 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.556596    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.556692 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.556667    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9cjt\" (UniqueName: \"kubernetes.io/projected/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-kube-api-access-l9cjt\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.556915 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.556888    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.556996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.556931    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.556996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.556981    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.558895 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.558875    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 29 10:52:07.559040 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.559024    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 29 10:52:07.559184 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.559166    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 29 10:52:07.559271 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.559166    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2lkn9\""
May 29 10:52:07.577431 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.577410    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.579493 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.579460    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 29 10:52:07.579595 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.579516    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2jqmr\""
May 29 10:52:07.579736 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.579721    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 29 10:52:07.579788 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.579721    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 29 10:52:07.657383 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657349    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0859261f-38fc-47b2-ae8f-23f946e87d42-metrics-client-ca\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.657567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657386    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmsgr\" (UniqueName: \"kubernetes.io/projected/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-api-access-jmsgr\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.657567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657411    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0859261f-38fc-47b2-ae8f-23f946e87d42-sys\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.657567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657433    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.657567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657484    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l9cjt\" (UniqueName: \"kubernetes.io/projected/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-kube-api-access-l9cjt\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.657717 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657572    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-textfile\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.657717 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657598    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-tls\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.657717 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657626    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.657717 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657653    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.657717 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657698    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.657934 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657728    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.657934 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657763    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.657934 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657792    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-accelerators-collector-config\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.657934 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657822    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.657934 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657848    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-wtmp\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.657934 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657875    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0859261f-38fc-47b2-ae8f-23f946e87d42-root\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.657934 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657902    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.657934 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657924    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.658360 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.657970    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddgs\" (UniqueName: \"kubernetes.io/projected/0859261f-38fc-47b2-ae8f-23f946e87d42-kube-api-access-rddgs\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.658551 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.658531    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.660155 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.660121    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.660285 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.660270    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.665414 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.665388    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9cjt\" (UniqueName: \"kubernetes.io/projected/f340ea9b-0c5c-45d2-b0f6-db1dc01354cc-kube-api-access-l9cjt\") pod \"openshift-state-metrics-9c959559f-k2hcg\" (UID: \"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.758468 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758384    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-tls\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.758468 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758427    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.758468 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758454    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.758767 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758488    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.758767 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758637    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-accelerators-collector-config\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.758767 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758673    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.758767 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758702    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-wtmp\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.758767 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758732    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0859261f-38fc-47b2-ae8f-23f946e87d42-root\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.758767 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758768    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.759119 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758823    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rddgs\" (UniqueName: \"kubernetes.io/projected/0859261f-38fc-47b2-ae8f-23f946e87d42-kube-api-access-rddgs\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759119 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758877    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0859261f-38fc-47b2-ae8f-23f946e87d42-metrics-client-ca\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759119 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758896    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.759119 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758905    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jmsgr\" (UniqueName: \"kubernetes.io/projected/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-api-access-jmsgr\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.759119 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758933    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0859261f-38fc-47b2-ae8f-23f946e87d42-sys\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759119 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758958    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759119 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758988    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-textfile\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759488 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.759265    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0859261f-38fc-47b2-ae8f-23f946e87d42-root\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759488 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.759277    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-textfile\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759488 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.759333    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.759488 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.759342    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-accelerators-collector-config\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759488 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.758877    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-wtmp\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759488 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.759395    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.759488 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.759412    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0859261f-38fc-47b2-ae8f-23f946e87d42-sys\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.759854 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.759719    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0859261f-38fc-47b2-ae8f-23f946e87d42-metrics-client-ca\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.761516 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.761489    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.761656 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.761638    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.761717 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.761674    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.762160 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.762136    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0859261f-38fc-47b2-ae8f-23f946e87d42-node-exporter-tls\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.775935 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.775916    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmsgr\" (UniqueName: \"kubernetes.io/projected/60068c98-1803-4dc6-8bc6-b17d7a9fa16e-kube-api-access-jmsgr\") pod \"kube-state-metrics-54fd6bc5c4-grsf9\" (UID: \"60068c98-1803-4dc6-8bc6-b17d7a9fa16e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.776132 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.776113    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddgs\" (UniqueName: \"kubernetes.io/projected/0859261f-38fc-47b2-ae8f-23f946e87d42-kube-api-access-rddgs\") pod \"node-exporter-llk2k\" (UID: \"0859261f-38fc-47b2-ae8f-23f946e87d42\") " pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.841038 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.840997    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"
May 29 10:52:07.866063 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.866033    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"
May 29 10:52:07.886017 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.885987    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-llk2k"
May 29 10:52:07.897232 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:52:07.897193    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0859261f_38fc_47b2_ae8f_23f946e87d42.slice/crio-53a9a178cef6f10e1913ff0f0c7e9252e8ac9aebd1042de4925fae1dac186925 WatchSource:0}: Error finding container 53a9a178cef6f10e1913ff0f0c7e9252e8ac9aebd1042de4925fae1dac186925: Status 404 returned error can't find the container with id 53a9a178cef6f10e1913ff0f0c7e9252e8ac9aebd1042de4925fae1dac186925
May 29 10:52:07.978616 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:07.978580    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg"]
May 29 10:52:07.981339 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:52:07.981311    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf340ea9b_0c5c_45d2_b0f6_db1dc01354cc.slice/crio-3b8269d5be1e96c51fc5685dabfc46a671f010c7696679e166a914bb7662025a WatchSource:0}: Error finding container 3b8269d5be1e96c51fc5685dabfc46a671f010c7696679e166a914bb7662025a: Status 404 returned error can't find the container with id 3b8269d5be1e96c51fc5685dabfc46a671f010c7696679e166a914bb7662025a
May 29 10:52:08.001877 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.001849    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9"]
May 29 10:52:08.004541 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:52:08.004515    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60068c98_1803_4dc6_8bc6_b17d7a9fa16e.slice/crio-93539557a9e3e370ca16742d543634bebbdee1f98ab45178630d19a33b8c23f8 WatchSource:0}: Error finding container 93539557a9e3e370ca16742d543634bebbdee1f98ab45178630d19a33b8c23f8: Status 404 returned error can't find the container with id 93539557a9e3e370ca16742d543634bebbdee1f98ab45178630d19a33b8c23f8
May 29 10:52:08.086302 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.086273    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9" event={"ID":"60068c98-1803-4dc6-8bc6-b17d7a9fa16e","Type":"ContainerStarted","Data":"93539557a9e3e370ca16742d543634bebbdee1f98ab45178630d19a33b8c23f8"}
May 29 10:52:08.087686 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.087661    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg" event={"ID":"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc","Type":"ContainerStarted","Data":"fdec781615a740e4933640d49a7ed2ec2480a68bfe77d526fb3cb40d2eaf0591"}
May 29 10:52:08.087783 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.087692    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg" event={"ID":"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc","Type":"ContainerStarted","Data":"3b8269d5be1e96c51fc5685dabfc46a671f010c7696679e166a914bb7662025a"}
May 29 10:52:08.088580 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.088560    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-llk2k" event={"ID":"0859261f-38fc-47b2-ae8f-23f946e87d42","Type":"ContainerStarted","Data":"53a9a178cef6f10e1913ff0f0c7e9252e8ac9aebd1042de4925fae1dac186925"}
May 29 10:52:08.600753 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.600719    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 29 10:52:08.630297 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.630265    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 29 10:52:08.630757 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.630729    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.635410 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.634673    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 29 10:52:08.635410 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.634691    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mffqx\""
May 29 10:52:08.635410 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.634751    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 29 10:52:08.635410 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.634856    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 29 10:52:08.635410 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.634964    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 29 10:52:08.635410 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.635005    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 29 10:52:08.635841 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.635822    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 29 10:52:08.636057 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.636019    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 29 10:52:08.636425 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.636214    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 29 10:52:08.636425 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.636271    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 29 10:52:08.665603 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665567    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-config-volume\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665773 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665612    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6fe6de1b-b96c-486e-8511-ef57291c1233-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665773 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665670    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6fe6de1b-b96c-486e-8511-ef57291c1233-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665773 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665720    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-web-config\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665773 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665748    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fe6de1b-b96c-486e-8511-ef57291c1233-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665779    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665805    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6fe6de1b-b96c-486e-8511-ef57291c1233-config-out\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665821    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665888    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe6de1b-b96c-486e-8511-ef57291c1233-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665919    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp6tn\" (UniqueName: \"kubernetes.io/projected/6fe6de1b-b96c-486e-8511-ef57291c1233-kube-api-access-xp6tn\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.665996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.665975    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.666276 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.666019    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.666276 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.666044    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.766609 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.766572    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xp6tn\" (UniqueName: \"kubernetes.io/projected/6fe6de1b-b96c-486e-8511-ef57291c1233-kube-api-access-xp6tn\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.766890 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.766632    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.766890 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.766684    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767078 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.766895    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767078 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.766986    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-config-volume\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767193 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.767083    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6fe6de1b-b96c-486e-8511-ef57291c1233-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767193 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.767126    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6fe6de1b-b96c-486e-8511-ef57291c1233-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767193 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.767173    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-web-config\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767374 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.767197    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fe6de1b-b96c-486e-8511-ef57291c1233-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767374 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.767234    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767374 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.767275    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6fe6de1b-b96c-486e-8511-ef57291c1233-config-out\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767374 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.767301    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767374 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.767348    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe6de1b-b96c-486e-8511-ef57291c1233-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.767600 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:52:08.767514    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fe6de1b-b96c-486e-8511-ef57291c1233-alertmanager-trusted-ca-bundle podName:6fe6de1b-b96c-486e-8511-ef57291c1233 nodeName:}" failed. No retries permitted until 2026-05-29 10:52:09.267480328 +0000 UTC m=+108.144042557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6fe6de1b-b96c-486e-8511-ef57291c1233-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6fe6de1b-b96c-486e-8511-ef57291c1233") : configmap references non-existent config key: ca-bundle.crt
May 29 10:52:08.769449 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.769348    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6fe6de1b-b96c-486e-8511-ef57291c1233-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.769653 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.769633    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fe6de1b-b96c-486e-8511-ef57291c1233-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.770901 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.770874    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.771951 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.771907    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.772569 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.772325    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.772569 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.772511    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6fe6de1b-b96c-486e-8511-ef57291c1233-config-out\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.772569 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.772520    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.772780 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.772665    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.772867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.772848    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-web-config\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.773392 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.773363    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6fe6de1b-b96c-486e-8511-ef57291c1233-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.774842 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.774814    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6fe6de1b-b96c-486e-8511-ef57291c1233-config-volume\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:08.783264 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:08.783199    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp6tn\" (UniqueName: \"kubernetes.io/projected/6fe6de1b-b96c-486e-8511-ef57291c1233-kube-api-access-xp6tn\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:09.092941 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.092904    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg" event={"ID":"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc","Type":"ContainerStarted","Data":"9e73d122f6ac39019765ea93522c3d87f7581d43142506e7dfe64a2fbb5fa4e2"}
May 29 10:52:09.272310 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.272279    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe6de1b-b96c-486e-8511-ef57291c1233-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:09.273149 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.273123    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe6de1b-b96c-486e-8511-ef57291c1233-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6fe6de1b-b96c-486e-8511-ef57291c1233\") " pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:09.543058 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.543027    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 29 10:52:09.590775 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.590740    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-d5d8f647b-stcr9"]
May 29 10:52:09.617679 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.617646    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d5d8f647b-stcr9"]
May 29 10:52:09.618057 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.617756    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.620311 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.620287    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 29 10:52:09.620432 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.620289    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 29 10:52:09.620432 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.620290    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 29 10:52:09.620607 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.620590    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fum83slll3g6q\""
May 29 10:52:09.620667 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.620657    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 29 10:52:09.620723 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.620668    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 29 10:52:09.620723 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.620680    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xntt9\""
May 29 10:52:09.675316 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.675279    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.675492 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.675325    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-grpc-tls\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.675492 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.675418    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-tls\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.675492 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.675460    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.675492 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.675482    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.675673 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.675568    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/258ae73c-dce5-4991-8562-cdf47bd22f50-metrics-client-ca\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.675673 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.675614    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxvk\" (UniqueName: \"kubernetes.io/projected/258ae73c-dce5-4991-8562-cdf47bd22f50-kube-api-access-5dxvk\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.675784 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.675673    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.776527 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.776487    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.776722 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.776541    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.776722 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.776562    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-grpc-tls\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.776722 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.776607    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-tls\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.776722 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.776639    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.776722 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.776664    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.776722 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.776713    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/258ae73c-dce5-4991-8562-cdf47bd22f50-metrics-client-ca\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.777040 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.776742    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxvk\" (UniqueName: \"kubernetes.io/projected/258ae73c-dce5-4991-8562-cdf47bd22f50-kube-api-access-5dxvk\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.777645 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.777616    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/258ae73c-dce5-4991-8562-cdf47bd22f50-metrics-client-ca\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.779749 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.779677    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-tls\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.779749 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.779682    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.779749 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.779729    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.780133 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.780089    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.780275 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.780181    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-grpc-tls\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.780391 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.780368    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/258ae73c-dce5-4991-8562-cdf47bd22f50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.784989 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.784960    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxvk\" (UniqueName: \"kubernetes.io/projected/258ae73c-dce5-4991-8562-cdf47bd22f50-kube-api-access-5dxvk\") pod \"thanos-querier-d5d8f647b-stcr9\" (UID: \"258ae73c-dce5-4991-8562-cdf47bd22f50\") " pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:09.927154 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:09.927071    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:10.472002 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:10.471712    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 29 10:52:10.477480 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:52:10.477437    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe6de1b_b96c_486e_8511_ef57291c1233.slice/crio-046b0a340eade45b4eb1b110b43f710da24340073cb61eb4bb289da29ba3df33 WatchSource:0}: Error finding container 046b0a340eade45b4eb1b110b43f710da24340073cb61eb4bb289da29ba3df33: Status 404 returned error can't find the container with id 046b0a340eade45b4eb1b110b43f710da24340073cb61eb4bb289da29ba3df33
May 29 10:52:10.485905 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:10.485881    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d5d8f647b-stcr9"]
May 29 10:52:10.488861 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:52:10.488836    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258ae73c_dce5_4991_8562_cdf47bd22f50.slice/crio-db51c50d3371b95f63bcc9a5c7a23c940109f01aaa13c9c7298d5ad43c41ad67 WatchSource:0}: Error finding container db51c50d3371b95f63bcc9a5c7a23c940109f01aaa13c9c7298d5ad43c41ad67: Status 404 returned error can't find the container with id db51c50d3371b95f63bcc9a5c7a23c940109f01aaa13c9c7298d5ad43c41ad67
May 29 10:52:11.102044 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.101999    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9" event={"ID":"258ae73c-dce5-4991-8562-cdf47bd22f50","Type":"ContainerStarted","Data":"db51c50d3371b95f63bcc9a5c7a23c940109f01aaa13c9c7298d5ad43c41ad67"}
May 29 10:52:11.104986 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.104910    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9" event={"ID":"60068c98-1803-4dc6-8bc6-b17d7a9fa16e","Type":"ContainerStarted","Data":"dd3a5026aa243a039c862ef2422f96291033a8597c992daf93f2b93ccd06f82e"}
May 29 10:52:11.104986 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.104950    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9" event={"ID":"60068c98-1803-4dc6-8bc6-b17d7a9fa16e","Type":"ContainerStarted","Data":"c4cfeb9200d6215ca53c05a6491d68a4a4f7bc1ad3ee2d7fda9f97942e1b3ec8"}
May 29 10:52:11.104986 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.104968    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9" event={"ID":"60068c98-1803-4dc6-8bc6-b17d7a9fa16e","Type":"ContainerStarted","Data":"32e26802e171f699f06ca9b12721a4719569c1340def016df71a4a2c53983e18"}
May 29 10:52:11.107210 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.107172    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg" event={"ID":"f340ea9b-0c5c-45d2-b0f6-db1dc01354cc","Type":"ContainerStarted","Data":"b9f784d6280698c628fac5da67144fb8fa6dea39da3f3f59242eb0f4765dae61"}
May 29 10:52:11.108698 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.108673    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6fe6de1b-b96c-486e-8511-ef57291c1233","Type":"ContainerStarted","Data":"046b0a340eade45b4eb1b110b43f710da24340073cb61eb4bb289da29ba3df33"}
May 29 10:52:11.110539 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.110512    2547 generic.go:358] "Generic (PLEG): container finished" podID="0859261f-38fc-47b2-ae8f-23f946e87d42" containerID="7beae5d05e61bec6c3ee27ade28ec4881212bbfabfb23c035d61a836e7f5cf1d" exitCode=0
May 29 10:52:11.110648 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.110550    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-llk2k" event={"ID":"0859261f-38fc-47b2-ae8f-23f946e87d42","Type":"ContainerDied","Data":"7beae5d05e61bec6c3ee27ade28ec4881212bbfabfb23c035d61a836e7f5cf1d"}
May 29 10:52:11.122349 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.122125    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-grsf9" podStartSLOduration=1.812047338 podStartE2EDuration="4.122109082s" podCreationTimestamp="2026-05-29 10:52:07 +0000 UTC" firstStartedPulling="2026-05-29 10:52:08.00638424 +0000 UTC m=+106.882946446" lastFinishedPulling="2026-05-29 10:52:10.316445965 +0000 UTC m=+109.193008190" observedRunningTime="2026-05-29 10:52:11.12176776 +0000 UTC m=+109.998329990" watchObservedRunningTime="2026-05-29 10:52:11.122109082 +0000 UTC m=+109.998671311"
May 29 10:52:11.138561 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:11.138507    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-k2hcg" podStartSLOduration=2.014873731 podStartE2EDuration="4.138489849s" podCreationTimestamp="2026-05-29 10:52:07 +0000 UTC" firstStartedPulling="2026-05-29 10:52:08.194555519 +0000 UTC m=+107.071117725" lastFinishedPulling="2026-05-29 10:52:10.318171631 +0000 UTC m=+109.194733843" observedRunningTime="2026-05-29 10:52:11.138149198 +0000 UTC m=+110.014711428" watchObservedRunningTime="2026-05-29 10:52:11.138489849 +0000 UTC m=+110.015052077"
May 29 10:52:12.116300 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.116171    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-llk2k" event={"ID":"0859261f-38fc-47b2-ae8f-23f946e87d42","Type":"ContainerStarted","Data":"518d297213624f4a244bd6581b9d52c22e39deb3bcb9754d497ae3ba7285a1ae"}
May 29 10:52:12.116300 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.116221    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-llk2k" event={"ID":"0859261f-38fc-47b2-ae8f-23f946e87d42","Type":"ContainerStarted","Data":"63ff6d7b5512c8e61b218909a289646b96d7e071320d0ce56438a9223da32dd7"}
May 29 10:52:12.137395 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.137333    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-llk2k" podStartSLOduration=2.746921831 podStartE2EDuration="5.137315261s" podCreationTimestamp="2026-05-29 10:52:07 +0000 UTC" firstStartedPulling="2026-05-29 10:52:07.899999413 +0000 UTC m=+106.776561620" lastFinishedPulling="2026-05-29 10:52:10.29039283 +0000 UTC m=+109.166955050" observedRunningTime="2026-05-29 10:52:12.136162335 +0000 UTC m=+111.012724564" watchObservedRunningTime="2026-05-29 10:52:12.137315261 +0000 UTC m=+111.013877492"
May 29 10:52:12.321904 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.321869    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-67cd48757d-j2pbl"]
May 29 10:52:12.349402 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.349374    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.354014 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.353949    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67cd48757d-j2pbl"]
May 29 10:52:12.363401 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.363369    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"]
May 29 10:52:12.385544 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.385472    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"]
May 29 10:52:12.385702 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.385612    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"
May 29 10:52:12.388499 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.388474    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 29 10:52:12.388777 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.388758    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-d6slw\""
May 29 10:52:12.399378 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.399354    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-service-ca\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.399504 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.399408    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-trusted-ca-bundle\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.399604 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.399514    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-oauth-config\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.399604 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.399571    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f0c431e-a55d-4dff-8673-5d6e870b4bed-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-69ww8\" (UID: \"5f0c431e-a55d-4dff-8673-5d6e870b4bed\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"
May 29 10:52:12.399710 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.399633    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-serving-cert\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.399710 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.399668    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxhkj\" (UniqueName: \"kubernetes.io/projected/46a67601-5911-4162-aa2f-b8c42347d521-kube-api-access-zxhkj\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.399710 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.399688    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-console-config\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.399710 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.399706    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-oauth-serving-cert\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.501070 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501034    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-oauth-config\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.501215 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501081    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f0c431e-a55d-4dff-8673-5d6e870b4bed-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-69ww8\" (UID: \"5f0c431e-a55d-4dff-8673-5d6e870b4bed\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"
May 29 10:52:12.501215 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501101    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-serving-cert\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.501215 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501120    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhkj\" (UniqueName: \"kubernetes.io/projected/46a67601-5911-4162-aa2f-b8c42347d521-kube-api-access-zxhkj\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.501215 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501141    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-console-config\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.501215 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501157    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-oauth-serving-cert\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.501215 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501186    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-service-ca\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.501215 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501212    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-trusted-ca-bundle\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.501617 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:52:12.501183    2547 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 29 10:52:12.501617 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:52:12.501332    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0c431e-a55d-4dff-8673-5d6e870b4bed-monitoring-plugin-cert podName:5f0c431e-a55d-4dff-8673-5d6e870b4bed nodeName:}" failed. No retries permitted until 2026-05-29 10:52:13.001309237 +0000 UTC m=+111.877871453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5f0c431e-a55d-4dff-8673-5d6e870b4bed-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-69ww8" (UID: "5f0c431e-a55d-4dff-8673-5d6e870b4bed") : secret "monitoring-plugin-cert" not found
May 29 10:52:12.501949 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501925    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-service-ca\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.502030 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.501972    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-oauth-serving-cert\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.502092 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.502077    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-trusted-ca-bundle\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.502160 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.502138    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-console-config\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.503453 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.503430    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-oauth-config\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.503593 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.503578    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-serving-cert\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.515579 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.514229    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxhkj\" (UniqueName: \"kubernetes.io/projected/46a67601-5911-4162-aa2f-b8c42347d521-kube-api-access-zxhkj\") pod \"console-67cd48757d-j2pbl\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") " pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.660429 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.660396    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:12.784568 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:12.784538    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67cd48757d-j2pbl"]
May 29 10:52:12.942601 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:52:12.942528    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a67601_5911_4162_aa2f_b8c42347d521.slice/crio-2cfaba789cb4ba93108f91b5509acf6d2bdc9c6d133536fa37951f93896c2c94 WatchSource:0}: Error finding container 2cfaba789cb4ba93108f91b5509acf6d2bdc9c6d133536fa37951f93896c2c94: Status 404 returned error can't find the container with id 2cfaba789cb4ba93108f91b5509acf6d2bdc9c6d133536fa37951f93896c2c94
May 29 10:52:13.005394 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.005365    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f0c431e-a55d-4dff-8673-5d6e870b4bed-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-69ww8\" (UID: \"5f0c431e-a55d-4dff-8673-5d6e870b4bed\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"
May 29 10:52:13.007567 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.007549    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f0c431e-a55d-4dff-8673-5d6e870b4bed-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-69ww8\" (UID: \"5f0c431e-a55d-4dff-8673-5d6e870b4bed\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"
May 29 10:52:13.121417 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.121387    2547 generic.go:358] "Generic (PLEG): container finished" podID="6fe6de1b-b96c-486e-8511-ef57291c1233" containerID="accb49a3257f3247a49920b0dac990a48aedf9afd21d385a179125c74c6f4962" exitCode=0
May 29 10:52:13.121805 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.121468    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6fe6de1b-b96c-486e-8511-ef57291c1233","Type":"ContainerDied","Data":"accb49a3257f3247a49920b0dac990a48aedf9afd21d385a179125c74c6f4962"}
May 29 10:52:13.123090 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.122985    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9" event={"ID":"258ae73c-dce5-4991-8562-cdf47bd22f50","Type":"ContainerStarted","Data":"aad0633ed4566a10affbd63a9b5102be5ddc81891e3b881cddaafbbc995157d4"}
May 29 10:52:13.124436 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.124415    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67cd48757d-j2pbl" event={"ID":"46a67601-5911-4162-aa2f-b8c42347d521","Type":"ContainerStarted","Data":"20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239"}
May 29 10:52:13.124641 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.124462    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67cd48757d-j2pbl" event={"ID":"46a67601-5911-4162-aa2f-b8c42347d521","Type":"ContainerStarted","Data":"2cfaba789cb4ba93108f91b5509acf6d2bdc9c6d133536fa37951f93896c2c94"}
May 29 10:52:13.162700 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.162650    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-67cd48757d-j2pbl" podStartSLOduration=1.162629543 podStartE2EDuration="1.162629543s" podCreationTimestamp="2026-05-29 10:52:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:52:13.162136884 +0000 UTC m=+112.038699115" watchObservedRunningTime="2026-05-29 10:52:13.162629543 +0000 UTC m=+112.039191769"
May 29 10:52:13.297050 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.297027    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"
May 29 10:52:13.422389 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:13.422357    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"]
May 29 10:52:13.426815 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:52:13.426782    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0c431e_a55d_4dff_8673_5d6e870b4bed.slice/crio-729c8451cc66b440c00c825fc983b4f2dbeb3ee828a4153200bdc11b15843b43 WatchSource:0}: Error finding container 729c8451cc66b440c00c825fc983b4f2dbeb3ee828a4153200bdc11b15843b43: Status 404 returned error can't find the container with id 729c8451cc66b440c00c825fc983b4f2dbeb3ee828a4153200bdc11b15843b43
May 29 10:52:14.131170 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:14.131099    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9" event={"ID":"258ae73c-dce5-4991-8562-cdf47bd22f50","Type":"ContainerStarted","Data":"21c35d202f8a4c7ebc8f16678b666665073d9efe974f14b29fa5fc877c4fe524"}
May 29 10:52:14.131170 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:14.131143    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9" event={"ID":"258ae73c-dce5-4991-8562-cdf47bd22f50","Type":"ContainerStarted","Data":"8a3b0d056c2858bdea89d2444cf64919c3bb81b9d2269fe20a643dc29c31fc94"}
May 29 10:52:14.132981 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:14.132935    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8" event={"ID":"5f0c431e-a55d-4dff-8673-5d6e870b4bed","Type":"ContainerStarted","Data":"729c8451cc66b440c00c825fc983b4f2dbeb3ee828a4153200bdc11b15843b43"}
May 29 10:52:16.140127 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.140097    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8" event={"ID":"5f0c431e-a55d-4dff-8673-5d6e870b4bed","Type":"ContainerStarted","Data":"2684c7f2b972cbc69ab4c4c5b4df71aa3ac39ad9818330241f77dea219c675a7"}
May 29 10:52:16.140577 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.140272    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"
May 29 10:52:16.142780 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.142754    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6fe6de1b-b96c-486e-8511-ef57291c1233","Type":"ContainerStarted","Data":"1e8ccf980b5f5c9d6d46dc0e6fca697984abe58ee4aabf6b7ec29a8399b63ca0"}
May 29 10:52:16.142912 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.142787    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6fe6de1b-b96c-486e-8511-ef57291c1233","Type":"ContainerStarted","Data":"ab77426a9f572b676f0fd76727418b2d417c657f582c6d2f953360d496f7469d"}
May 29 10:52:16.142912 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.142802    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6fe6de1b-b96c-486e-8511-ef57291c1233","Type":"ContainerStarted","Data":"64113d6fe06f1bd57604319e7e4c272c94144ae60601bbace5dde9e7dcea50d7"}
May 29 10:52:16.145819 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.145799    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8"
May 29 10:52:16.145989 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.145967    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9" event={"ID":"258ae73c-dce5-4991-8562-cdf47bd22f50","Type":"ContainerStarted","Data":"41eaeec6166f13a8617c8f580e5689e3334ea95715c7a81a99a6fb117d2225bc"}
May 29 10:52:16.146044 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.145997    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9" event={"ID":"258ae73c-dce5-4991-8562-cdf47bd22f50","Type":"ContainerStarted","Data":"11718b3f34851d3b583b7dade7b6c7e1499aaa432291b81f1f70977e1029e750"}
May 29 10:52:16.146044 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.146009    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9" event={"ID":"258ae73c-dce5-4991-8562-cdf47bd22f50","Type":"ContainerStarted","Data":"627c33b18fefb7bc61ed5b3499e1ea648ad17cec530c5bacd2408373b8d60296"}
May 29 10:52:16.146184 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.146170    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:16.156679 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.156574    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-69ww8" podStartSLOduration=1.8963347929999999 podStartE2EDuration="4.156557902s" podCreationTimestamp="2026-05-29 10:52:12 +0000 UTC" firstStartedPulling="2026-05-29 10:52:13.429705925 +0000 UTC m=+112.306268131" lastFinishedPulling="2026-05-29 10:52:15.68992902 +0000 UTC m=+114.566491240" observedRunningTime="2026-05-29 10:52:16.154348318 +0000 UTC m=+115.030910571" watchObservedRunningTime="2026-05-29 10:52:16.156557902 +0000 UTC m=+115.033120131"
May 29 10:52:16.189093 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:16.189038    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9" podStartSLOduration=1.993834053 podStartE2EDuration="7.189018866s" podCreationTimestamp="2026-05-29 10:52:09 +0000 UTC" firstStartedPulling="2026-05-29 10:52:10.490722328 +0000 UTC m=+109.367284534" lastFinishedPulling="2026-05-29 10:52:15.685907141 +0000 UTC m=+114.562469347" observedRunningTime="2026-05-29 10:52:16.18773977 +0000 UTC m=+115.064302012" watchObservedRunningTime="2026-05-29 10:52:16.189018866 +0000 UTC m=+115.065581095"
May 29 10:52:17.152343 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:17.152295    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6fe6de1b-b96c-486e-8511-ef57291c1233","Type":"ContainerStarted","Data":"a5f9c0b5cc5e0b13a1d6e8ce80ebb6540094f954db0b16af4cb0d5c7be9b63d1"}
May 29 10:52:17.152343 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:17.152347    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6fe6de1b-b96c-486e-8511-ef57291c1233","Type":"ContainerStarted","Data":"e53c210ed0ac8468e3be5920b3a10c8ddc80155d5352978f7b919ed8f7684f0e"}
May 29 10:52:17.152750 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:17.152361    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6fe6de1b-b96c-486e-8511-ef57291c1233","Type":"ContainerStarted","Data":"e9f1d8a58fc2f5f7fc7592942b98487dcff949c29006c08e494d78c4dd0f2329"}
May 29 10:52:17.181817 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:17.181760    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.975300324 podStartE2EDuration="9.181741333s" podCreationTimestamp="2026-05-29 10:52:08 +0000 UTC" firstStartedPulling="2026-05-29 10:52:10.479464024 +0000 UTC m=+109.356026231" lastFinishedPulling="2026-05-29 10:52:15.685905027 +0000 UTC m=+114.562467240" observedRunningTime="2026-05-29 10:52:17.179755134 +0000 UTC m=+116.056317364" watchObservedRunningTime="2026-05-29 10:52:17.181741333 +0000 UTC m=+116.058303572"
May 29 10:52:22.158211 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:22.158184    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-d5d8f647b-stcr9"
May 29 10:52:22.660658 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:22.660619    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:22.660861 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:22.660753    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:22.666608 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:22.666584    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:23.173753 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:23.173724    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:52:23.219805 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:23.219765    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f656d898-b7qlm"]
May 29 10:52:48.242017 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.241954    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-56f656d898-b7qlm" podUID="ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" containerName="console" containerID="cri-o://5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4" gracePeriod=15
May 29 10:52:48.479715 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.479691    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56f656d898-b7qlm_ab6bb824-3dc1-4020-a75d-d5a78fa07ec1/console/0.log"
May 29 10:52:48.479846 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.479753    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:52:48.528223 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528191    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-config\") pod \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") "
May 29 10:52:48.528404 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528255    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-oauth-config\") pod \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") "
May 29 10:52:48.528404 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528303    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-serving-cert\") pod \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") "
May 29 10:52:48.528404 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528363    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65tk8\" (UniqueName: \"kubernetes.io/projected/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-kube-api-access-65tk8\") pod \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") "
May 29 10:52:48.528404 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528396    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-service-ca\") pod \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") "
May 29 10:52:48.528611 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528422    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-trusted-ca-bundle\") pod \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") "
May 29 10:52:48.528611 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528462    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-oauth-serving-cert\") pod \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\" (UID: \"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1\") "
May 29 10:52:48.528711 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528607    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-config" (OuterVolumeSpecName: "console-config") pod "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" (UID: "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:52:48.528852 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528827    2547 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:52:48.528989 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528884    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-service-ca" (OuterVolumeSpecName: "service-ca") pod "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" (UID: "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:52:48.528989 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528896    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" (UID: "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:52:48.528989 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.528942    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" (UID: "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:52:48.530625 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.530602    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" (UID: "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:52:48.530729 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.530678    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-kube-api-access-65tk8" (OuterVolumeSpecName: "kube-api-access-65tk8") pod "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" (UID: "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1"). InnerVolumeSpecName "kube-api-access-65tk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:52:48.530729 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.530708    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" (UID: "ab6bb824-3dc1-4020-a75d-d5a78fa07ec1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:52:48.629346 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.629298    2547 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-oauth-serving-cert\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:52:48.629346 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.629341    2547 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-oauth-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:52:48.629346 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.629351    2547 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-console-serving-cert\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:52:48.629346 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.629359    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65tk8\" (UniqueName: \"kubernetes.io/projected/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-kube-api-access-65tk8\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:52:48.629651 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.629371    2547 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:52:48.629651 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:48.629380    2547 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1-trusted-ca-bundle\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:52:49.245531 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.245504    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56f656d898-b7qlm_ab6bb824-3dc1-4020-a75d-d5a78fa07ec1/console/0.log"
May 29 10:52:49.245938 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.245542    2547 generic.go:358] "Generic (PLEG): container finished" podID="ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" containerID="5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4" exitCode=2
May 29 10:52:49.245938 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.245589    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f656d898-b7qlm" event={"ID":"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1","Type":"ContainerDied","Data":"5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4"}
May 29 10:52:49.245938 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.245609    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f656d898-b7qlm"
May 29 10:52:49.245938 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.245622    2547 scope.go:117] "RemoveContainer" containerID="5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4"
May 29 10:52:49.245938 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.245612    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f656d898-b7qlm" event={"ID":"ab6bb824-3dc1-4020-a75d-d5a78fa07ec1","Type":"ContainerDied","Data":"8593748ecd6c90e160b460bc46766027df81a46ccd2cda875b1bac07c1d44842"}
May 29 10:52:49.254849 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.254832    2547 scope.go:117] "RemoveContainer" containerID="5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4"
May 29 10:52:49.255101 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:52:49.255086    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4\": container with ID starting with 5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4 not found: ID does not exist" containerID="5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4"
May 29 10:52:49.255150 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.255110    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4"} err="failed to get container status \"5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4\": rpc error: code = NotFound desc = could not find container \"5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4\": container with ID starting with 5e03947171e3d47fefe609ba4703f02be4a5a90e995015b107fe6c00abc140f4 not found: ID does not exist"
May 29 10:52:49.267806 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.267785    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f656d898-b7qlm"]
May 29 10:52:49.271372 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.271349    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56f656d898-b7qlm"]
May 29 10:52:49.716507 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:52:49.716473    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" path="/var/lib/kubelet/pods/ab6bb824-3dc1-4020-a75d-d5a78fa07ec1/volumes"
May 29 10:53:32.330552 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.330510    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"]
May 29 10:53:32.331032 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.330833    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" containerName="console"
May 29 10:53:32.331032 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.330844    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" containerName="console"
May 29 10:53:32.331032 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.330905    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="ab6bb824-3dc1-4020-a75d-d5a78fa07ec1" containerName="console"
May 29 10:53:32.335356 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.335335    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.339123 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.339098    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 29 10:53:32.339123 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.339139    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 29 10:53:32.339343 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.339193    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 29 10:53:32.339571 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.339558    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 29 10:53:32.339782 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.339768    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-29vph\""
May 29 10:53:32.339867 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.339851    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 29 10:53:32.344913 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.344893    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 29 10:53:32.368111 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.368083    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"]
May 29 10:53:32.504465 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.504427    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.504647 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.504481    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-metrics-client-ca\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.504647 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.504512    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-telemeter-client-tls\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.504647 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.504551    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zllqz\" (UniqueName: \"kubernetes.io/projected/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-kube-api-access-zllqz\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.504751 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.504637    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-federate-client-tls\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.504787 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.504748    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-secret-telemeter-client\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.504787 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.504775    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-serving-certs-ca-bundle\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.504851 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.504803    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.606137 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.606039    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-federate-client-tls\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.606313 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.606192    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-secret-telemeter-client\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.606313 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.606215    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-serving-certs-ca-bundle\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.606313 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.606234    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.606313 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.606285    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.606470 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.606350    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-metrics-client-ca\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.606470 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.606398    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-telemeter-client-tls\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.606470 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.606432    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zllqz\" (UniqueName: \"kubernetes.io/projected/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-kube-api-access-zllqz\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.607033 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.607002    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-serving-certs-ca-bundle\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.607448 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.607421    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-metrics-client-ca\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.607448 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.607436    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.608829 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.608792    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-secret-telemeter-client\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.608928 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.608845    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-telemeter-client-tls\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.609166 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.609146    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-federate-client-tls\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.609203 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.609164    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.617360 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.617338    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zllqz\" (UniqueName: \"kubernetes.io/projected/ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b-kube-api-access-zllqz\") pod \"telemeter-client-859cccc5c4-lc5sn\" (UID: \"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b\") " pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.645406 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.645369    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"
May 29 10:53:32.811407 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:32.811369    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-859cccc5c4-lc5sn"]
May 29 10:53:32.816172 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:53:32.816146    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4cd5e5_96e0_4c9d_8a0f_77f4e39a1b8b.slice/crio-af24b936aeb9cb2db02ebf9d5b37ab206bfae7892f1f1f57e7d40f19bca34526 WatchSource:0}: Error finding container af24b936aeb9cb2db02ebf9d5b37ab206bfae7892f1f1f57e7d40f19bca34526: Status 404 returned error can't find the container with id af24b936aeb9cb2db02ebf9d5b37ab206bfae7892f1f1f57e7d40f19bca34526
May 29 10:53:33.375347 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:33.375310    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn" event={"ID":"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b","Type":"ContainerStarted","Data":"af24b936aeb9cb2db02ebf9d5b37ab206bfae7892f1f1f57e7d40f19bca34526"}
May 29 10:53:35.383750 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:35.383712    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn" event={"ID":"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b","Type":"ContainerStarted","Data":"fd30193c8698267886f580413e099989d2fb09953be0f5524e46e3a15032f38c"}
May 29 10:53:35.383750 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:35.383751    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn" event={"ID":"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b","Type":"ContainerStarted","Data":"2ba334f74ff35caf356dda02ffb5d70e44280236793b8e94cbcb85e26a3c08b9"}
May 29 10:53:35.383750 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:35.383761    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn" event={"ID":"ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b","Type":"ContainerStarted","Data":"416d5fcd187cd3bf77a4c3abde93b5db68d3f5165e9be7c3fdaa1036cab1584d"}
May 29 10:53:35.428736 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:35.428672    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-859cccc5c4-lc5sn" podStartSLOduration=1.553950737 podStartE2EDuration="3.428654179s" podCreationTimestamp="2026-05-29 10:53:32 +0000 UTC" firstStartedPulling="2026-05-29 10:53:32.817890432 +0000 UTC m=+191.694452637" lastFinishedPulling="2026-05-29 10:53:34.692593869 +0000 UTC m=+193.569156079" observedRunningTime="2026-05-29 10:53:35.425943797 +0000 UTC m=+194.302506026" watchObservedRunningTime="2026-05-29 10:53:35.428654179 +0000 UTC m=+194.305216408"
May 29 10:53:36.247022 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.246989    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d95c58dc-xlnxf"]
May 29 10:53:36.250374 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.250346    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.253256 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.253212    2547 status_manager.go:895] "Failed to get status for pod" podUID="a07b5993-c2d1-4d23-acb4-13c4a26e0352" pod="openshift-console/console-57d95c58dc-xlnxf" err="pods \"console-57d95c58dc-xlnxf\" is forbidden: User \"system:node:ip-10-0-142-12.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-12.ec2.internal' and this object"
May 29 10:53:36.288947 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.288917    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d95c58dc-xlnxf"]
May 29 10:53:36.342035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.341998    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-oauth-config\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.342035 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.342041    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-serving-cert\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.342285 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.342112    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-trusted-ca-bundle\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.342285 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.342138    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44m95\" (UniqueName: \"kubernetes.io/projected/a07b5993-c2d1-4d23-acb4-13c4a26e0352-kube-api-access-44m95\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.342285 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.342157    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-oauth-serving-cert\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.342285 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.342180    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-config\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.342409 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.342297    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-service-ca\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.442883 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.442839    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-trusted-ca-bundle\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.442883 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.442876    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44m95\" (UniqueName: \"kubernetes.io/projected/a07b5993-c2d1-4d23-acb4-13c4a26e0352-kube-api-access-44m95\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.443477 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.443019    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-oauth-serving-cert\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.443477 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.443093    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-config\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.443477 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.443149    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-service-ca\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.443477 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.443313    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-oauth-config\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.443477 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.443391    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-serving-cert\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.443939 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.443908    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-oauth-serving-cert\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.444063 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.444043    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-service-ca\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.444130 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.444058    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-trusted-ca-bundle\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.444186 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.444163    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-config\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.446089 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.446063    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-oauth-config\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.446187 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.446072    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-serving-cert\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.471514 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.471481    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44m95\" (UniqueName: \"kubernetes.io/projected/a07b5993-c2d1-4d23-acb4-13c4a26e0352-kube-api-access-44m95\") pod \"console-57d95c58dc-xlnxf\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") " pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.558963 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.558916    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:36.766091 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:36.766066    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d95c58dc-xlnxf"]
May 29 10:53:36.767702 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:53:36.767668    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07b5993_c2d1_4d23_acb4_13c4a26e0352.slice/crio-720e9e210232dce81034f767e3a64e9ca8d34a537b1a9de5720d892c9511d7c1 WatchSource:0}: Error finding container 720e9e210232dce81034f767e3a64e9ca8d34a537b1a9de5720d892c9511d7c1: Status 404 returned error can't find the container with id 720e9e210232dce81034f767e3a64e9ca8d34a537b1a9de5720d892c9511d7c1
May 29 10:53:37.393636 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:37.393601    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d95c58dc-xlnxf" event={"ID":"a07b5993-c2d1-4d23-acb4-13c4a26e0352","Type":"ContainerStarted","Data":"ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232"}
May 29 10:53:37.393636 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:37.393637    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d95c58dc-xlnxf" event={"ID":"a07b5993-c2d1-4d23-acb4-13c4a26e0352","Type":"ContainerStarted","Data":"720e9e210232dce81034f767e3a64e9ca8d34a537b1a9de5720d892c9511d7c1"}
May 29 10:53:37.422624 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:37.422568    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-57d95c58dc-xlnxf" podStartSLOduration=1.422554111 podStartE2EDuration="1.422554111s" podCreationTimestamp="2026-05-29 10:53:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:53:37.421724068 +0000 UTC m=+196.298286296" watchObservedRunningTime="2026-05-29 10:53:37.422554111 +0000 UTC m=+196.299116339"
May 29 10:53:46.559850 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:46.559810    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:46.559850 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:46.559858    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:46.564649 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:46.564625    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:47.431308 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.431274    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:53:47.517377 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.517346    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67cd48757d-j2pbl"]
May 29 10:53:47.782057 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.782026    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b7cb4cdd7-2r9qs"]
May 29 10:53:47.786419 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.786395    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.836619 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.836586    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnmxs\" (UniqueName: \"kubernetes.io/projected/45c11cc2-d291-49d3-86df-1c2e0b258508-kube-api-access-hnmxs\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.836804 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.836642    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-serving-cert\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.836804 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.836699    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-oauth-serving-cert\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.836804 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.836734    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-service-ca\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.836804 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.836760    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-trusted-ca-bundle\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.836804 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.836794    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-oauth-config\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.837010 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.836821    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-console-config\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.877702 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.877670    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b7cb4cdd7-2r9qs"]
May 29 10:53:47.937743 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.937710    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-oauth-serving-cert\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.937743 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.937745    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-service-ca\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.937999 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.937768    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-trusted-ca-bundle\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.937999 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.937910    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-oauth-config\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.937999 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.937950    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-console-config\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.938156 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.938029    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hnmxs\" (UniqueName: \"kubernetes.io/projected/45c11cc2-d291-49d3-86df-1c2e0b258508-kube-api-access-hnmxs\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.938156 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.938080    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-serving-cert\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.938582 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.938548    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-oauth-serving-cert\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.938723 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.938617    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-service-ca\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.938723 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.938687    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-trusted-ca-bundle\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.938829 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.938735    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-console-config\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.940441 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.940424    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-serving-cert\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.940487 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.940451    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-oauth-config\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:47.957542 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:47.957510    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnmxs\" (UniqueName: \"kubernetes.io/projected/45c11cc2-d291-49d3-86df-1c2e0b258508-kube-api-access-hnmxs\") pod \"console-6b7cb4cdd7-2r9qs\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") " pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:48.095739 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:48.095651    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:48.281923 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:48.281892    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b7cb4cdd7-2r9qs"]
May 29 10:53:48.284987 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:53:48.284956    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c11cc2_d291_49d3_86df_1c2e0b258508.slice/crio-9bbd14394627c7257d37d57d50144965afc95847d36977af902cb715816d45b4 WatchSource:0}: Error finding container 9bbd14394627c7257d37d57d50144965afc95847d36977af902cb715816d45b4: Status 404 returned error can't find the container with id 9bbd14394627c7257d37d57d50144965afc95847d36977af902cb715816d45b4
May 29 10:53:48.435042 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:48.434952    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7cb4cdd7-2r9qs" event={"ID":"45c11cc2-d291-49d3-86df-1c2e0b258508","Type":"ContainerStarted","Data":"9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec"}
May 29 10:53:48.435042 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:48.435001    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7cb4cdd7-2r9qs" event={"ID":"45c11cc2-d291-49d3-86df-1c2e0b258508","Type":"ContainerStarted","Data":"9bbd14394627c7257d37d57d50144965afc95847d36977af902cb715816d45b4"}
May 29 10:53:58.096723 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:58.096668    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:58.096723 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:58.096731    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:58.101517 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:58.101496    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:58.133380 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:58.133328    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6b7cb4cdd7-2r9qs" podStartSLOduration=11.133311847 podStartE2EDuration="11.133311847s" podCreationTimestamp="2026-05-29 10:53:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:53:48.478498581 +0000 UTC m=+207.355060809" watchObservedRunningTime="2026-05-29 10:53:58.133311847 +0000 UTC m=+217.009874076"
May 29 10:53:58.470377 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:58.470301    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 10:53:58.546332 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:53:58.546303    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d95c58dc-xlnxf"]
May 29 10:54:12.540486 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.540425    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-67cd48757d-j2pbl" podUID="46a67601-5911-4162-aa2f-b8c42347d521" containerName="console" containerID="cri-o://20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239" gracePeriod=15
May 29 10:54:12.799147 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.799085    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67cd48757d-j2pbl_46a67601-5911-4162-aa2f-b8c42347d521/console/0.log"
May 29 10:54:12.799147 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.799146    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:54:12.965640 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.965607    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-oauth-config\") pod \"46a67601-5911-4162-aa2f-b8c42347d521\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") "
May 29 10:54:12.965834 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.965672    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-serving-cert\") pod \"46a67601-5911-4162-aa2f-b8c42347d521\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") "
May 29 10:54:12.965912 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.965858    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-oauth-serving-cert\") pod \"46a67601-5911-4162-aa2f-b8c42347d521\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") "
May 29 10:54:12.965912 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.965898    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxhkj\" (UniqueName: \"kubernetes.io/projected/46a67601-5911-4162-aa2f-b8c42347d521-kube-api-access-zxhkj\") pod \"46a67601-5911-4162-aa2f-b8c42347d521\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") "
May 29 10:54:12.966018 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.965968    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-service-ca\") pod \"46a67601-5911-4162-aa2f-b8c42347d521\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") "
May 29 10:54:12.966070 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.966019    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-console-config\") pod \"46a67601-5911-4162-aa2f-b8c42347d521\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") "
May 29 10:54:12.966119 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.966074    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-trusted-ca-bundle\") pod \"46a67601-5911-4162-aa2f-b8c42347d521\" (UID: \"46a67601-5911-4162-aa2f-b8c42347d521\") "
May 29 10:54:12.966222 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.966189    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "46a67601-5911-4162-aa2f-b8c42347d521" (UID: "46a67601-5911-4162-aa2f-b8c42347d521"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:54:12.966359 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.966344    2547 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-oauth-serving-cert\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:12.966483 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.966458    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-console-config" (OuterVolumeSpecName: "console-config") pod "46a67601-5911-4162-aa2f-b8c42347d521" (UID: "46a67601-5911-4162-aa2f-b8c42347d521"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:54:12.966529 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.966464    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-service-ca" (OuterVolumeSpecName: "service-ca") pod "46a67601-5911-4162-aa2f-b8c42347d521" (UID: "46a67601-5911-4162-aa2f-b8c42347d521"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:54:12.966713 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.966691    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "46a67601-5911-4162-aa2f-b8c42347d521" (UID: "46a67601-5911-4162-aa2f-b8c42347d521"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:54:12.968048 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.968011    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a67601-5911-4162-aa2f-b8c42347d521-kube-api-access-zxhkj" (OuterVolumeSpecName: "kube-api-access-zxhkj") pod "46a67601-5911-4162-aa2f-b8c42347d521" (UID: "46a67601-5911-4162-aa2f-b8c42347d521"). InnerVolumeSpecName "kube-api-access-zxhkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:54:12.968367 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.968337    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "46a67601-5911-4162-aa2f-b8c42347d521" (UID: "46a67601-5911-4162-aa2f-b8c42347d521"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:54:12.968367 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:12.968350    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "46a67601-5911-4162-aa2f-b8c42347d521" (UID: "46a67601-5911-4162-aa2f-b8c42347d521"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:54:13.067852 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.067762    2547 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-trusted-ca-bundle\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:13.067852 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.067796    2547 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-oauth-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:13.067852 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.067807    2547 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46a67601-5911-4162-aa2f-b8c42347d521-console-serving-cert\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:13.067852 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.067816    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxhkj\" (UniqueName: \"kubernetes.io/projected/46a67601-5911-4162-aa2f-b8c42347d521-kube-api-access-zxhkj\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:13.067852 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.067827    2547 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:13.067852 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.067836    2547 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46a67601-5911-4162-aa2f-b8c42347d521-console-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:13.509433 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.509358    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67cd48757d-j2pbl_46a67601-5911-4162-aa2f-b8c42347d521/console/0.log"
May 29 10:54:13.509433 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.509396    2547 generic.go:358] "Generic (PLEG): container finished" podID="46a67601-5911-4162-aa2f-b8c42347d521" containerID="20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239" exitCode=2
May 29 10:54:13.509612 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.509491    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67cd48757d-j2pbl"
May 29 10:54:13.509612 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.509489    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67cd48757d-j2pbl" event={"ID":"46a67601-5911-4162-aa2f-b8c42347d521","Type":"ContainerDied","Data":"20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239"}
May 29 10:54:13.509612 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.509526    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67cd48757d-j2pbl" event={"ID":"46a67601-5911-4162-aa2f-b8c42347d521","Type":"ContainerDied","Data":"2cfaba789cb4ba93108f91b5509acf6d2bdc9c6d133536fa37951f93896c2c94"}
May 29 10:54:13.509612 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.509542    2547 scope.go:117] "RemoveContainer" containerID="20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239"
May 29 10:54:13.517773 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.517756    2547 scope.go:117] "RemoveContainer" containerID="20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239"
May 29 10:54:13.518029 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:54:13.518009    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239\": container with ID starting with 20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239 not found: ID does not exist" containerID="20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239"
May 29 10:54:13.518067 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.518039    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239"} err="failed to get container status \"20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239\": rpc error: code = NotFound desc = could not find container \"20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239\": container with ID starting with 20f0b04839345603e976cbc49306c02415eba5793ac4cb683251e492a157e239 not found: ID does not exist"
May 29 10:54:13.532861 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.532836    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67cd48757d-j2pbl"]
May 29 10:54:13.537856 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.537833    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67cd48757d-j2pbl"]
May 29 10:54:13.717141 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:13.717110    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46a67601-5911-4162-aa2f-b8c42347d521" path="/var/lib/kubelet/pods/46a67601-5911-4162-aa2f-b8c42347d521/volumes"
May 29 10:54:23.565165 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.565100    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-57d95c58dc-xlnxf" podUID="a07b5993-c2d1-4d23-acb4-13c4a26e0352" containerName="console" containerID="cri-o://ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232" gracePeriod=15
May 29 10:54:23.796544 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.796523    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57d95c58dc-xlnxf_a07b5993-c2d1-4d23-acb4-13c4a26e0352/console/0.log"
May 29 10:54:23.796656 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.796581    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:54:23.856110 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856037    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44m95\" (UniqueName: \"kubernetes.io/projected/a07b5993-c2d1-4d23-acb4-13c4a26e0352-kube-api-access-44m95\") pod \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") "
May 29 10:54:23.856110 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856076    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-serving-cert\") pod \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") "
May 29 10:54:23.856110 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856106    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-oauth-config\") pod \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") "
May 29 10:54:23.856381 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856125    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-config\") pod \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") "
May 29 10:54:23.856381 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856166    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-trusted-ca-bundle\") pod \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") "
May 29 10:54:23.856381 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856194    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-oauth-serving-cert\") pod \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") "
May 29 10:54:23.856381 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856269    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-service-ca\") pod \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\" (UID: \"a07b5993-c2d1-4d23-acb4-13c4a26e0352\") "
May 29 10:54:23.856625 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856596    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-config" (OuterVolumeSpecName: "console-config") pod "a07b5993-c2d1-4d23-acb4-13c4a26e0352" (UID: "a07b5993-c2d1-4d23-acb4-13c4a26e0352"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:54:23.856722 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856645    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a07b5993-c2d1-4d23-acb4-13c4a26e0352" (UID: "a07b5993-c2d1-4d23-acb4-13c4a26e0352"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:54:23.856775 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856752    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a07b5993-c2d1-4d23-acb4-13c4a26e0352" (UID: "a07b5993-c2d1-4d23-acb4-13c4a26e0352"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:54:23.856823 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.856804    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-service-ca" (OuterVolumeSpecName: "service-ca") pod "a07b5993-c2d1-4d23-acb4-13c4a26e0352" (UID: "a07b5993-c2d1-4d23-acb4-13c4a26e0352"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 10:54:23.858212 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.858194    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a07b5993-c2d1-4d23-acb4-13c4a26e0352" (UID: "a07b5993-c2d1-4d23-acb4-13c4a26e0352"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:54:23.858599 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.858578    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07b5993-c2d1-4d23-acb4-13c4a26e0352-kube-api-access-44m95" (OuterVolumeSpecName: "kube-api-access-44m95") pod "a07b5993-c2d1-4d23-acb4-13c4a26e0352" (UID: "a07b5993-c2d1-4d23-acb4-13c4a26e0352"). InnerVolumeSpecName "kube-api-access-44m95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:54:23.858704 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.858633    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a07b5993-c2d1-4d23-acb4-13c4a26e0352" (UID: "a07b5993-c2d1-4d23-acb4-13c4a26e0352"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:54:23.957753 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.957718    2547 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:23.957753 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.957745    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44m95\" (UniqueName: \"kubernetes.io/projected/a07b5993-c2d1-4d23-acb4-13c4a26e0352-kube-api-access-44m95\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:23.957753 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.957756    2547 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-serving-cert\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:23.957967 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.957766    2547 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-oauth-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:23.957967 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.957774    2547 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-console-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:23.957967 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.957784    2547 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-trusted-ca-bundle\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:23.957967 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:23.957793    2547 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a07b5993-c2d1-4d23-acb4-13c4a26e0352-oauth-serving-cert\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:54:24.542164 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.542137    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57d95c58dc-xlnxf_a07b5993-c2d1-4d23-acb4-13c4a26e0352/console/0.log"
May 29 10:54:24.542364 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.542181    2547 generic.go:358] "Generic (PLEG): container finished" podID="a07b5993-c2d1-4d23-acb4-13c4a26e0352" containerID="ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232" exitCode=2
May 29 10:54:24.542364 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.542255    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d95c58dc-xlnxf"
May 29 10:54:24.542364 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.542280    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d95c58dc-xlnxf" event={"ID":"a07b5993-c2d1-4d23-acb4-13c4a26e0352","Type":"ContainerDied","Data":"ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232"}
May 29 10:54:24.542364 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.542322    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d95c58dc-xlnxf" event={"ID":"a07b5993-c2d1-4d23-acb4-13c4a26e0352","Type":"ContainerDied","Data":"720e9e210232dce81034f767e3a64e9ca8d34a537b1a9de5720d892c9511d7c1"}
May 29 10:54:24.542364 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.542340    2547 scope.go:117] "RemoveContainer" containerID="ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232"
May 29 10:54:24.550774 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.550760    2547 scope.go:117] "RemoveContainer" containerID="ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232"
May 29 10:54:24.551022 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:54:24.551004    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232\": container with ID starting with ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232 not found: ID does not exist" containerID="ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232"
May 29 10:54:24.551068 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.551031    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232"} err="failed to get container status \"ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232\": rpc error: code = NotFound desc = could not find container \"ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232\": container with ID starting with ca59c79df8dc99e1c0b38c77dbbd83d0c41927c85e072d7b802e4c71c1ad3232 not found: ID does not exist"
May 29 10:54:24.573554 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.573526    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d95c58dc-xlnxf"]
May 29 10:54:24.577077 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:24.577058    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57d95c58dc-xlnxf"]
May 29 10:54:25.716483 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:54:25.716448    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a07b5993-c2d1-4d23-acb4-13c4a26e0352" path="/var/lib/kubelet/pods/a07b5993-c2d1-4d23-acb4-13c4a26e0352/volumes"
May 29 10:55:21.642346 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:55:21.642317    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 10:55:21.642885 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:55:21.642389    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 10:55:21.647011 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:55:21.646995    2547 kubelet.go:1628] "Image garbage collection succeeded"
May 29 10:56:09.190178 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.190100    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"]
May 29 10:56:09.190573 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.190429    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a07b5993-c2d1-4d23-acb4-13c4a26e0352" containerName="console"
May 29 10:56:09.190573 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.190440    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a07b5993-c2d1-4d23-acb4-13c4a26e0352" containerName="console"
May 29 10:56:09.190573 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.190457    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a67601-5911-4162-aa2f-b8c42347d521" containerName="console"
May 29 10:56:09.190573 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.190462    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a67601-5911-4162-aa2f-b8c42347d521" containerName="console"
May 29 10:56:09.190573 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.190514    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="a07b5993-c2d1-4d23-acb4-13c4a26e0352" containerName="console"
May 29 10:56:09.190573 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.190524    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="46a67601-5911-4162-aa2f-b8c42347d521" containerName="console"
May 29 10:56:09.192277 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.192261    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"
May 29 10:56:09.195883 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.195857    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 29 10:56:09.196039 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.195924    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 29 10:56:09.196654 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.196640    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-jtd77\""
May 29 10:56:09.268725 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.268693    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"]
May 29 10:56:09.323131 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.323084    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl66s\" (UniqueName: \"kubernetes.io/projected/e7b90bb3-5012-4d14-9986-7a44ee5e2141-kube-api-access-fl66s\") pod \"feast-operator-controller-manager-6fccb7dc4-zf6lg\" (UID: \"e7b90bb3-5012-4d14-9986-7a44ee5e2141\") " pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"
May 29 10:56:09.424004 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.423965    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fl66s\" (UniqueName: \"kubernetes.io/projected/e7b90bb3-5012-4d14-9986-7a44ee5e2141-kube-api-access-fl66s\") pod \"feast-operator-controller-manager-6fccb7dc4-zf6lg\" (UID: \"e7b90bb3-5012-4d14-9986-7a44ee5e2141\") " pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"
May 29 10:56:09.436109 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.436075    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl66s\" (UniqueName: \"kubernetes.io/projected/e7b90bb3-5012-4d14-9986-7a44ee5e2141-kube-api-access-fl66s\") pod \"feast-operator-controller-manager-6fccb7dc4-zf6lg\" (UID: \"e7b90bb3-5012-4d14-9986-7a44ee5e2141\") " pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"
May 29 10:56:09.502551 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.502457    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"
May 29 10:56:09.648000 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:56:09.647968    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b90bb3_5012_4d14_9986_7a44ee5e2141.slice/crio-4215cbd61b5c7bcf94e2ff15ac5a1561312ecbbf60862ccb9baf197ea4b7cb0f WatchSource:0}: Error finding container 4215cbd61b5c7bcf94e2ff15ac5a1561312ecbbf60862ccb9baf197ea4b7cb0f: Status 404 returned error can't find the container with id 4215cbd61b5c7bcf94e2ff15ac5a1561312ecbbf60862ccb9baf197ea4b7cb0f
May 29 10:56:09.649807 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.649793    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 10:56:09.679618 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.679581    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"]
May 29 10:56:09.838019 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:09.837979    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg" event={"ID":"e7b90bb3-5012-4d14-9986-7a44ee5e2141","Type":"ContainerStarted","Data":"4215cbd61b5c7bcf94e2ff15ac5a1561312ecbbf60862ccb9baf197ea4b7cb0f"}
May 29 10:56:12.848831 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:12.848790    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg" event={"ID":"e7b90bb3-5012-4d14-9986-7a44ee5e2141","Type":"ContainerStarted","Data":"bb1190d9a5850f0fa80ae84351d7cb64018d42edf80408af0dd1e70b36fdfc0b"}
May 29 10:56:12.849214 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:12.848916    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"
May 29 10:56:12.878650 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:12.878587    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg" podStartSLOduration=1.597001477 podStartE2EDuration="3.878569943s" podCreationTimestamp="2026-05-29 10:56:09 +0000 UTC" firstStartedPulling="2026-05-29 10:56:09.649923371 +0000 UTC m=+348.526485577" lastFinishedPulling="2026-05-29 10:56:11.931491832 +0000 UTC m=+350.808054043" observedRunningTime="2026-05-29 10:56:12.876328579 +0000 UTC m=+351.752890808" watchObservedRunningTime="2026-05-29 10:56:12.878569943 +0000 UTC m=+351.755132175"
May 29 10:56:23.854745 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:23.854714    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"
May 29 10:56:52.069292 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.069250    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"]
May 29 10:56:52.077397 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.077371    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.081705 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.081668    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 29 10:56:52.081840 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.081782    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 29 10:56:52.081923 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.081904    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 10:56:52.082032 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.082018    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 29 10:56:52.082103 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.082068    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-qszrn\""
May 29 10:56:52.082156 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.082112    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 29 10:56:52.082211 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.082168    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 29 10:56:52.082996 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.082980    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 10:56:52.090394 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.090371    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"]
May 29 10:56:52.165168 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.165127    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-online-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.165356 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.165177    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgsr\" (UniqueName: \"kubernetes.io/projected/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-kube-api-access-vfgsr\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.165356 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.165296    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-registry-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.165356 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.165320    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-feast-data\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.165356 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.165345    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-offline-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.165492 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.165367    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-ui-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.266388 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.266344    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-online-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.266594 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.266414    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgsr\" (UniqueName: \"kubernetes.io/projected/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-kube-api-access-vfgsr\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.266594 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.266471    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-registry-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.266594 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.266492    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-feast-data\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.266594 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.266524    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-offline-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.266594 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.266548    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-ui-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.266858 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:56:52.266654    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 29 10:56:52.266858 ip-10-0-142-12 kubenswrapper[2547]: E0529 10:56:52.266744    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-ui-tls podName:fae9bca4-3ec0-48d7-b21d-eb271588b8b7 nodeName:}" failed. No retries permitted until 2026-05-29 10:56:52.766721694 +0000 UTC m=+391.643283920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-ui-tls") pod "feast-simple-feast-setup-76b886f884-wth6p" (UID: "fae9bca4-3ec0-48d7-b21d-eb271588b8b7") : secret "feast-simple-feast-setup-ui-tls" not found
May 29 10:56:52.266974 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.266927    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-feast-data\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.268917 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.268895    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-registry-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.268917 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.268921    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-online-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.269107 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.268964    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-offline-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.277221 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.277196    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgsr\" (UniqueName: \"kubernetes.io/projected/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-kube-api-access-vfgsr\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.770320 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.770284    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-ui-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.772585 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.772564    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-ui-tls\") pod \"feast-simple-feast-setup-76b886f884-wth6p\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:52.999001 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:52.998955    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:56:53.132815 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:53.132757    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"]
May 29 10:56:53.134950 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:56:53.134921    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfae9bca4_3ec0_48d7_b21d_eb271588b8b7.slice/crio-83a0fab4b755560b69a18821f0ef4134de767b2664037fc9218ec75ba8ce6edf WatchSource:0}: Error finding container 83a0fab4b755560b69a18821f0ef4134de767b2664037fc9218ec75ba8ce6edf: Status 404 returned error can't find the container with id 83a0fab4b755560b69a18821f0ef4134de767b2664037fc9218ec75ba8ce6edf
May 29 10:56:53.965923 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:56:53.965880    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerStarted","Data":"83a0fab4b755560b69a18821f0ef4134de767b2664037fc9218ec75ba8ce6edf"}
May 29 10:57:21.056987 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:21.056947    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerStarted","Data":"851927da62d443e0fa170f8671c527a15dedf9f30005ab9a0422b7e2d7721ab6"}
May 29 10:57:26.073597 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:26.073563    2547 generic.go:358] "Generic (PLEG): container finished" podID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerID="851927da62d443e0fa170f8671c527a15dedf9f30005ab9a0422b7e2d7721ab6" exitCode=0
May 29 10:57:26.074007 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:26.073631    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerDied","Data":"851927da62d443e0fa170f8671c527a15dedf9f30005ab9a0422b7e2d7721ab6"}
May 29 10:57:27.078835 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:27.078796    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerStarted","Data":"c5ac17eef1766142c22d8c6b5e5527d6431ee537fbfdf0721558fca8816f5916"}
May 29 10:57:33.098321 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:33.098286    2547 generic.go:358] "Generic (PLEG): container finished" podID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerID="c5ac17eef1766142c22d8c6b5e5527d6431ee537fbfdf0721558fca8816f5916" exitCode=0
May 29 10:57:33.098798 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:33.098364    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerDied","Data":"c5ac17eef1766142c22d8c6b5e5527d6431ee537fbfdf0721558fca8816f5916"}
May 29 10:57:33.107959 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:33.107938    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 29 10:57:34.107753 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:34.107709    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerStarted","Data":"ad22a70895845c619a067db21fe895993aab6c98ca2bb4f4afa445e35bccc0fb"}
May 29 10:57:34.108258 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:34.107762    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerStarted","Data":"6a5cbc7e4ad3a6e5fd5cb1ba029056e36435c8e050cfaf1a5ddc5c29e7e15519"}
May 29 10:57:34.108258 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:34.107798    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerStarted","Data":"06130ca4089557279a2b3cf907399e38170b7865bd715ecf014bd14a0cc860d7"}
May 29 10:57:34.108258 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:34.107813    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerStarted","Data":"59337d995b3b1adb6b5b4f38afbb21fc73051b66cebf07a9ed0f4ed208995cd5"}
May 29 10:57:34.133954 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:34.133884    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podStartSLOduration=14.753200245 podStartE2EDuration="42.133862485s" podCreationTimestamp="2026-05-29 10:56:52 +0000 UTC" firstStartedPulling="2026-05-29 10:56:53.136605032 +0000 UTC m=+392.013167238" lastFinishedPulling="2026-05-29 10:57:20.517267267 +0000 UTC m=+419.393829478" observedRunningTime="2026-05-29 10:57:34.132062714 +0000 UTC m=+433.008624943" watchObservedRunningTime="2026-05-29 10:57:34.133862485 +0000 UTC m=+433.010424718"
May 29 10:57:35.000093 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:34.999998    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:35.000093 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:35.000049    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:35.000093 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:35.000063    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:35.000423 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:35.000214    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:35.002022 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:35.001883    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused"
May 29 10:57:35.002022 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:35.001921    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused"
May 29 10:57:35.002022 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:35.001958    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused"
May 29 10:57:35.002022 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:35.001924    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused"
May 29 10:57:37.999975 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:37.999919    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused"
May 29 10:57:37.999975 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:37.999961    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused"
May 29 10:57:38.000600 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:37.999913    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused"
May 29 10:57:38.000600 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:37.999913    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused"
May 29 10:57:41.000896 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.000864    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.001479 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.001321    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.001561 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.001507    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.001914 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.001844    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.001914 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.001881    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.001914 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.001895    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.002180 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.001993    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.002180 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.002121    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.002321 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.002305    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.005899 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.005878    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.132748 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.132723    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:57:41.135668 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:57:41.135654    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:58:43.388617 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.388585    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"]
May 29 10:58:43.389173 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.388917    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="registry" containerID="cri-o://59337d995b3b1adb6b5b4f38afbb21fc73051b66cebf07a9ed0f4ed208995cd5" gracePeriod=30
May 29 10:58:43.389173 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.388974    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="ui" containerID="cri-o://ad22a70895845c619a067db21fe895993aab6c98ca2bb4f4afa445e35bccc0fb" gracePeriod=30
May 29 10:58:43.389173 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.389001    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="online" containerID="cri-o://06130ca4089557279a2b3cf907399e38170b7865bd715ecf014bd14a0cc860d7" gracePeriod=30
May 29 10:58:43.389173 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.389001    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="offline" containerID="cri-o://6a5cbc7e4ad3a6e5fd5cb1ba029056e36435c8e050cfaf1a5ddc5c29e7e15519" gracePeriod=30
May 29 10:58:43.777892 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.777840    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"]
May 29 10:58:43.782855 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.782828    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:43.785344 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.785318    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-tvr5k\""
May 29 10:58:43.794649 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.794621    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"]
May 29 10:58:43.938946 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.938906    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8e26c107-9c4e-4c9e-b370-9fd368371ee5-feast-data\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:43.939098 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.938970    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-online-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:43.939098 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.939040    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbdb\" (UniqueName: \"kubernetes.io/projected/8e26c107-9c4e-4c9e-b370-9fd368371ee5-kube-api-access-7gbdb\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:43.939098 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.939068    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-ui-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:43.939098 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.939096    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-registry-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:43.939265 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:43.939178    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-offline-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.040195 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.040106    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8e26c107-9c4e-4c9e-b370-9fd368371ee5-feast-data\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.040195 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.040168    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-online-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.040195 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.040192    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbdb\" (UniqueName: \"kubernetes.io/projected/8e26c107-9c4e-4c9e-b370-9fd368371ee5-kube-api-access-7gbdb\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.040496 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.040209    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-ui-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.040496 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.040233    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-registry-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.040496 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.040295    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-offline-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.040636 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.040613    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8e26c107-9c4e-4c9e-b370-9fd368371ee5-feast-data\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.042870 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.042846    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-online-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.042983 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.042916    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-registry-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.042983 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.042961    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-ui-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.043096 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.043075    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-offline-tls\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.048897 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.048871    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbdb\" (UniqueName: \"kubernetes.io/projected/8e26c107-9c4e-4c9e-b370-9fd368371ee5-kube-api-access-7gbdb\") pod \"feast-simple-feast-setup-76b886f884-jk67k\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") " pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.092863 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.092825    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:44.232361 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.232322    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"]
May 29 10:58:44.235476 ip-10-0-142-12 kubenswrapper[2547]: W0529 10:58:44.235445    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e26c107_9c4e_4c9e_b370_9fd368371ee5.slice/crio-62cf860ec8ac6e66f410de3694eaa707fd8764e68c09ce63ca90eae6b7981507 WatchSource:0}: Error finding container 62cf860ec8ac6e66f410de3694eaa707fd8764e68c09ce63ca90eae6b7981507: Status 404 returned error can't find the container with id 62cf860ec8ac6e66f410de3694eaa707fd8764e68c09ce63ca90eae6b7981507
May 29 10:58:44.325931 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.325891    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerStarted","Data":"a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90"}
May 29 10:58:44.325931 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.325935    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerStarted","Data":"62cf860ec8ac6e66f410de3694eaa707fd8764e68c09ce63ca90eae6b7981507"}
May 29 10:58:44.328675 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.328652    2547 generic.go:358] "Generic (PLEG): container finished" podID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerID="ad22a70895845c619a067db21fe895993aab6c98ca2bb4f4afa445e35bccc0fb" exitCode=0
May 29 10:58:44.328675 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.328673    2547 generic.go:358] "Generic (PLEG): container finished" podID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerID="06130ca4089557279a2b3cf907399e38170b7865bd715ecf014bd14a0cc860d7" exitCode=0
May 29 10:58:44.328814 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.328701    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerDied","Data":"ad22a70895845c619a067db21fe895993aab6c98ca2bb4f4afa445e35bccc0fb"}
May 29 10:58:44.328814 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:44.328719    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerDied","Data":"06130ca4089557279a2b3cf907399e38170b7865bd715ecf014bd14a0cc860d7"}
May 29 10:58:49.346618 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:49.346582    2547 generic.go:358] "Generic (PLEG): container finished" podID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerID="a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90" exitCode=0
May 29 10:58:49.347106 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:49.346657    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerDied","Data":"a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90"}
May 29 10:58:50.351281 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:50.351221    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerStarted","Data":"f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d"}
May 29 10:58:51.002183 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:51.002132    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused"
May 29 10:58:51.133072 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:51.133031    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused"
May 29 10:58:55.368027 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:55.367993    2547 generic.go:358] "Generic (PLEG): container finished" podID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerID="f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d" exitCode=0
May 29 10:58:55.368415 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:55.368062    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerDied","Data":"f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d"}
May 29 10:58:56.375262 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:56.375201    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerStarted","Data":"390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5"}
May 29 10:58:56.375718 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:56.375275    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerStarted","Data":"e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633"}
May 29 10:58:56.375718 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:56.375292    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerStarted","Data":"db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb"}
May 29 10:58:56.375718 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:56.375305    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerStarted","Data":"914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e"}
May 29 10:58:56.401571 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:56.401509    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podStartSLOduration=13.401489046 podStartE2EDuration="13.401489046s" podCreationTimestamp="2026-05-29 10:58:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:58:56.398697968 +0000 UTC m=+515.275260199" watchObservedRunningTime="2026-05-29 10:58:56.401489046 +0000 UTC m=+515.278051276"
May 29 10:58:59.094075 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:59.093973    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:59.094652 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:59.094626    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:59.094796 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:59.094671    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:59.094796 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:59.094686    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:58:59.096257 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:59.096196    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 29 10:58:59.096369 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:59.096260    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 29 10:58:59.096369 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:59.096267    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused"
May 29 10:58:59.096482 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:58:59.096425    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused"
May 29 10:59:01.002347 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:01.002295    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused"
May 29 10:59:01.132893 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:01.132854    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused"
May 29 10:59:02.093937 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.093903    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.094425 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.094181    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.094559 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.094543    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.094930 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.094908    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.095074 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.094938    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.095074 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.094951    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.095193 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.095157    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.095267 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.095215    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.095371 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.095356    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.098404 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.098387    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.398776 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.398686    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:02.401687 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:02.401661    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 10:59:11.002685 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:11.002581    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused"
May 29 10:59:11.003091 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:11.002727    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:59:11.132931 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:11.132887    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused"
May 29 10:59:11.133085 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:11.133010    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:59:13.435317 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:13.435272    2547 generic.go:358] "Generic (PLEG): container finished" podID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerID="6a5cbc7e4ad3a6e5fd5cb1ba029056e36435c8e050cfaf1a5ddc5c29e7e15519" exitCode=137
May 29 10:59:13.435317 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:13.435316    2547 generic.go:358] "Generic (PLEG): container finished" podID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerID="59337d995b3b1adb6b5b4f38afbb21fc73051b66cebf07a9ed0f4ed208995cd5" exitCode=137
May 29 10:59:13.435663 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:13.435340    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerDied","Data":"6a5cbc7e4ad3a6e5fd5cb1ba029056e36435c8e050cfaf1a5ddc5c29e7e15519"}
May 29 10:59:13.435663 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:13.435372    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerDied","Data":"59337d995b3b1adb6b5b4f38afbb21fc73051b66cebf07a9ed0f4ed208995cd5"}
May 29 10:59:14.041093 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.041069    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:59:14.094384 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.094353    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-registry-tls\") pod \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") "
May 29 10:59:14.094521 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.094414    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-feast-data\") pod \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") "
May 29 10:59:14.094521 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.094481    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-online-tls\") pod \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") "
May 29 10:59:14.094521 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.094501    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-offline-tls\") pod \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") "
May 29 10:59:14.094521 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.094519    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-ui-tls\") pod \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") "
May 29 10:59:14.094728 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.094554    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfgsr\" (UniqueName: \"kubernetes.io/projected/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-kube-api-access-vfgsr\") pod \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\" (UID: \"fae9bca4-3ec0-48d7-b21d-eb271588b8b7\") "
May 29 10:59:14.094967 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.094935    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-feast-data" (OuterVolumeSpecName: "feast-data") pod "fae9bca4-3ec0-48d7-b21d-eb271588b8b7" (UID: "fae9bca4-3ec0-48d7-b21d-eb271588b8b7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 10:59:14.096670 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.096645    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-online-tls" (OuterVolumeSpecName: "online-tls") pod "fae9bca4-3ec0-48d7-b21d-eb271588b8b7" (UID: "fae9bca4-3ec0-48d7-b21d-eb271588b8b7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:59:14.096800 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.096726    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "fae9bca4-3ec0-48d7-b21d-eb271588b8b7" (UID: "fae9bca4-3ec0-48d7-b21d-eb271588b8b7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:59:14.096800 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.096748    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fae9bca4-3ec0-48d7-b21d-eb271588b8b7" (UID: "fae9bca4-3ec0-48d7-b21d-eb271588b8b7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:59:14.097020 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.096995    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "fae9bca4-3ec0-48d7-b21d-eb271588b8b7" (UID: "fae9bca4-3ec0-48d7-b21d-eb271588b8b7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 10:59:14.097109 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.097027    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-kube-api-access-vfgsr" (OuterVolumeSpecName: "kube-api-access-vfgsr") pod "fae9bca4-3ec0-48d7-b21d-eb271588b8b7" (UID: "fae9bca4-3ec0-48d7-b21d-eb271588b8b7"). InnerVolumeSpecName "kube-api-access-vfgsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 10:59:14.195742 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.195651    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:59:14.195742 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.195688    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:59:14.195742 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.195699    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:59:14.195742 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.195707    2547 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-ui-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:59:14.195742 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.195717    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgsr\" (UniqueName: \"kubernetes.io/projected/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-kube-api-access-vfgsr\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:59:14.195742 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.195728    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fae9bca4-3ec0-48d7-b21d-eb271588b8b7-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 10:59:14.441053 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.441023    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"
May 29 10:59:14.441475 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.441016    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p" event={"ID":"fae9bca4-3ec0-48d7-b21d-eb271588b8b7","Type":"ContainerDied","Data":"83a0fab4b755560b69a18821f0ef4134de767b2664037fc9218ec75ba8ce6edf"}
May 29 10:59:14.441475 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.441139    2547 scope.go:117] "RemoveContainer" containerID="ad22a70895845c619a067db21fe895993aab6c98ca2bb4f4afa445e35bccc0fb"
May 29 10:59:14.450128 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.450111    2547 scope.go:117] "RemoveContainer" containerID="6a5cbc7e4ad3a6e5fd5cb1ba029056e36435c8e050cfaf1a5ddc5c29e7e15519"
May 29 10:59:14.457975 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.457956    2547 scope.go:117] "RemoveContainer" containerID="06130ca4089557279a2b3cf907399e38170b7865bd715ecf014bd14a0cc860d7"
May 29 10:59:14.464890 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.464866    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"]
May 29 10:59:14.468500 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.468475    2547 scope.go:117] "RemoveContainer" containerID="59337d995b3b1adb6b5b4f38afbb21fc73051b66cebf07a9ed0f4ed208995cd5"
May 29 10:59:14.470368 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.470346    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-wth6p"]
May 29 10:59:14.476177 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.476156    2547 scope.go:117] "RemoveContainer" containerID="c5ac17eef1766142c22d8c6b5e5527d6431ee537fbfdf0721558fca8816f5916"
May 29 10:59:14.483106 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:14.483087    2547 scope.go:117] "RemoveContainer" containerID="851927da62d443e0fa170f8671c527a15dedf9f30005ab9a0422b7e2d7721ab6"
May 29 10:59:15.717306 ip-10-0-142-12 kubenswrapper[2547]: I0529 10:59:15.717271    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" path="/var/lib/kubelet/pods/fae9bca4-3ec0-48d7-b21d-eb271588b8b7/volumes"
May 29 11:00:05.756946 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.756909    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"]
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757260    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="feast-apply"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757275    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="feast-apply"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757297    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="registry"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757303    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="registry"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757312    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="feast-init"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757317    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="feast-init"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757325    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="offline"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757330    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="offline"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757339    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="online"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757345    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="online"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757353    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="ui"
May 29 11:00:05.757395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757359    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="ui"
May 29 11:00:05.757751 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757412    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="ui"
May 29 11:00:05.757751 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757421    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="online"
May 29 11:00:05.757751 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757428    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="offline"
May 29 11:00:05.757751 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.757435    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="fae9bca4-3ec0-48d7-b21d-eb271588b8b7" containerName="registry"
May 29 11:00:05.761065 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.761045    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.763599 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.763575    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 29 11:00:05.763719 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.763658    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 29 11:00:05.764259 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.764230    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 29 11:00:05.764678 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.764664    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 29 11:00:05.764933 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.764916    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 29 11:00:05.765029 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.764956    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 29 11:00:05.765179 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.765165    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 29 11:00:05.765341 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.765323    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-rf7s2\""
May 29 11:00:05.783488 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.783456    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"]
May 29 11:00:05.827591 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.827562    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-online-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.827773 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.827618    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-ui-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.827773 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.827672    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad1f5bf-3db0-454b-8726-f46a778b4c17-feast-data\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.827773 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.827700    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9ad1f5bf-3db0-454b-8726-f46a778b4c17-registry-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.827773 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.827763    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r669r\" (UniqueName: \"kubernetes.io/projected/9ad1f5bf-3db0-454b-8726-f46a778b4c17-kube-api-access-r669r\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.827930 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.827822    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-offline-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.928580 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.928540    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-offline-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.928580 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.928586    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-online-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.928810 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.928625    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-ui-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.928810 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.928651    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad1f5bf-3db0-454b-8726-f46a778b4c17-feast-data\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.928810 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.928694    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9ad1f5bf-3db0-454b-8726-f46a778b4c17-registry-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.928810 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.928727    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r669r\" (UniqueName: \"kubernetes.io/projected/9ad1f5bf-3db0-454b-8726-f46a778b4c17-kube-api-access-r669r\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.928810 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:00:05.928756    2547 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 29 11:00:05.929054 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:00:05.928828    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-ui-tls podName:9ad1f5bf-3db0-454b-8726-f46a778b4c17 nodeName:}" failed. No retries permitted until 2026-05-29 11:00:06.428805072 +0000 UTC m=+585.305367278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-ui-tls") pod "feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" (UID: "9ad1f5bf-3db0-454b-8726-f46a778b4c17") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 29 11:00:05.929215 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.929195    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad1f5bf-3db0-454b-8726-f46a778b4c17-feast-data\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.929486 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.929467    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9ad1f5bf-3db0-454b-8726-f46a778b4c17-registry-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.931040 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.931011    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-online-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.931146 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.931060    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-offline-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:05.938722 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:05.938691    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r669r\" (UniqueName: \"kubernetes.io/projected/9ad1f5bf-3db0-454b-8726-f46a778b4c17-kube-api-access-r669r\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:06.435974 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:06.435933    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-ui-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:06.438523 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:06.438488    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-ui-tls\") pod \"feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:06.670711 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:06.670663    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:06.812530 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:06.812505    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"]
May 29 11:00:06.813204 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:00:06.813177    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ad1f5bf_3db0_454b_8726_f46a778b4c17.slice/crio-d22262a36947e20d1e53fd90d669fc56bd912cec55060ae83bf2738ff1d90028 WatchSource:0}: Error finding container d22262a36947e20d1e53fd90d669fc56bd912cec55060ae83bf2738ff1d90028: Status 404 returned error can't find the container with id d22262a36947e20d1e53fd90d669fc56bd912cec55060ae83bf2738ff1d90028
May 29 11:00:07.604255 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:07.604200    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerStarted","Data":"c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad"}
May 29 11:00:07.604255 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:07.604257    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerStarted","Data":"d22262a36947e20d1e53fd90d669fc56bd912cec55060ae83bf2738ff1d90028"}
May 29 11:00:11.618669 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:11.618632    2547 generic.go:358] "Generic (PLEG): container finished" podID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerID="c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad" exitCode=0
May 29 11:00:11.619050 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:11.618687    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerDied","Data":"c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad"}
May 29 11:00:12.623635 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:12.623597    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerStarted","Data":"b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9"}
May 29 11:00:17.639855 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:17.639820    2547 generic.go:358] "Generic (PLEG): container finished" podID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerID="b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9" exitCode=0
May 29 11:00:17.640230 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:17.639886    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerDied","Data":"b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9"}
May 29 11:00:18.647392 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.647350    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerStarted","Data":"cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd"}
May 29 11:00:18.647873 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.647404    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerStarted","Data":"41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b"}
May 29 11:00:18.647873 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.647423    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerStarted","Data":"9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c"}
May 29 11:00:18.670931 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.670883    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:18.670931 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.670944    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:18.671162 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.670962    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:18.672824 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.672717    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podStartSLOduration=13.672699073 podStartE2EDuration="13.672699073s" podCreationTimestamp="2026-05-29 11:00:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:00:18.671876433 +0000 UTC m=+597.548438656" watchObservedRunningTime="2026-05-29 11:00:18.672699073 +0000 UTC m=+597.549261301"
May 29 11:00:18.672968 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.672827    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 29 11:00:18.672968 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.672845    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused"
May 29 11:00:18.672968 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:18.672848    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 29 11:00:21.670971 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:21.670934    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 29 11:00:21.671446 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:21.670939    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused"
May 29 11:00:21.671446 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:21.671052    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 29 11:00:21.675962 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:21.675933    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 11:00:21.676463 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:21.676440    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 11:00:24.671645 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:24.671609    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:24.672222 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:24.672202    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:24.672330 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:24.672234    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:24.672397 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:24.672383    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:24.676127 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:24.676113    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:25.670686 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:25.670653    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:25.670852 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:25.670696    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:25.671304 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:25.671284    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:00:25.673550 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:00:25.673531    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:01:27.927550 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:27.927518    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"]
May 29 11:01:27.928009 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:27.927852    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="online" containerID="cri-o://9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c" gracePeriod=30
May 29 11:01:27.928009 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:27.927939    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="ui" containerID="cri-o://cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd" gracePeriod=30
May 29 11:01:27.928009 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:27.927946    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="offline" containerID="cri-o://41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b" gracePeriod=30
May 29 11:01:28.048210 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.048174    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"]
May 29 11:01:28.048682 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.048598    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="registry" containerID="cri-o://914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e" gracePeriod=30
May 29 11:01:28.048682 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.048648    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="ui" containerID="cri-o://390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5" gracePeriod=30
May 29 11:01:28.048867 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.048644    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="offline" containerID="cri-o://e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633" gracePeriod=30
May 29 11:01:28.048867 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.048647    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="online" containerID="cri-o://db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb" gracePeriod=30
May 29 11:01:28.858607 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.858574    2547 generic.go:358] "Generic (PLEG): container finished" podID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerID="390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5" exitCode=0
May 29 11:01:28.858607 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.858600    2547 generic.go:358] "Generic (PLEG): container finished" podID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerID="db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb" exitCode=0
May 29 11:01:28.858843 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.858643    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerDied","Data":"390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5"}
May 29 11:01:28.858843 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.858676    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerDied","Data":"db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb"}
May 29 11:01:28.860847 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.860825    2547 generic.go:358] "Generic (PLEG): container finished" podID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerID="9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c" exitCode=0
May 29 11:01:28.860963 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:28.860858    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerDied","Data":"9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c"}
May 29 11:01:31.872635 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:31.872597    2547 generic.go:358] "Generic (PLEG): container finished" podID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerID="cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd" exitCode=0
May 29 11:01:31.873032 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:31.872665    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerDied","Data":"cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd"}
May 29 11:01:32.095221 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:32.095179    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 29 11:01:32.399767 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:32.399724    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 29 11:01:35.671777 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:35.671736    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 29 11:01:35.672181 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:35.671735    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 29 11:01:42.095895 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:42.095847    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 29 11:01:42.399275 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:42.399155    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 29 11:01:45.671495 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:45.671449    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 29 11:01:45.671900 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:45.671495    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 29 11:01:52.095412 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:52.095364    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 29 11:01:52.095877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:52.095559    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 11:01:52.399592 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:52.399497    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 29 11:01:52.399750 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:52.399638    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 11:01:55.670943 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:55.670893    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 29 11:01:55.670943 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:55.670923    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 29 11:01:55.671387 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:55.671015    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:01:55.671387 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:55.671095    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:01:58.147221 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.147201    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:01:58.187029 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.186969    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad1f5bf-3db0-454b-8726-f46a778b4c17-feast-data\") pod \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") "
May 29 11:01:58.187029 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.187007    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-online-tls\") pod \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") "
May 29 11:01:58.187211 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.187052    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9ad1f5bf-3db0-454b-8726-f46a778b4c17-registry-tls\") pod \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") "
May 29 11:01:58.187211 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.187076    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-ui-tls\") pod \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") "
May 29 11:01:58.187211 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.187109    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-offline-tls\") pod \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") "
May 29 11:01:58.187211 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.187177    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r669r\" (UniqueName: \"kubernetes.io/projected/9ad1f5bf-3db0-454b-8726-f46a778b4c17-kube-api-access-r669r\") pod \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\" (UID: \"9ad1f5bf-3db0-454b-8726-f46a778b4c17\") "
May 29 11:01:58.188075 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.188019    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad1f5bf-3db0-454b-8726-f46a778b4c17-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9ad1f5bf-3db0-454b-8726-f46a778b4c17" (UID: "9ad1f5bf-3db0-454b-8726-f46a778b4c17"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 11:01:58.188196 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.188096    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad1f5bf-3db0-454b-8726-f46a778b4c17-feast-data" (OuterVolumeSpecName: "feast-data") pod "9ad1f5bf-3db0-454b-8726-f46a778b4c17" (UID: "9ad1f5bf-3db0-454b-8726-f46a778b4c17"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:01:58.190111 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.190077    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9ad1f5bf-3db0-454b-8726-f46a778b4c17" (UID: "9ad1f5bf-3db0-454b-8726-f46a778b4c17"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:01:58.190221 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.190145    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "9ad1f5bf-3db0-454b-8726-f46a778b4c17" (UID: "9ad1f5bf-3db0-454b-8726-f46a778b4c17"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:01:58.191354 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.191079    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-online-tls" (OuterVolumeSpecName: "online-tls") pod "9ad1f5bf-3db0-454b-8726-f46a778b4c17" (UID: "9ad1f5bf-3db0-454b-8726-f46a778b4c17"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:01:58.196019 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.195991    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad1f5bf-3db0-454b-8726-f46a778b4c17-kube-api-access-r669r" (OuterVolumeSpecName: "kube-api-access-r669r") pod "9ad1f5bf-3db0-454b-8726-f46a778b4c17" (UID: "9ad1f5bf-3db0-454b-8726-f46a778b4c17"). InnerVolumeSpecName "kube-api-access-r669r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:01:58.289572 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.289533    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.289572 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.289565    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r669r\" (UniqueName: \"kubernetes.io/projected/9ad1f5bf-3db0-454b-8726-f46a778b4c17-kube-api-access-r669r\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.289572 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.289575    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad1f5bf-3db0-454b-8726-f46a778b4c17-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.289793 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.289587    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.289793 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.289594    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9ad1f5bf-3db0-454b-8726-f46a778b4c17-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.289793 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.289602    2547 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9ad1f5bf-3db0-454b-8726-f46a778b4c17-ui-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.686610 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.686585    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 11:01:58.793817 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.793786    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8e26c107-9c4e-4c9e-b370-9fd368371ee5-feast-data\") pod \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") "
May 29 11:01:58.793817 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.793822    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-online-tls\") pod \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") "
May 29 11:01:58.794050 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.793852    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-offline-tls\") pod \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") "
May 29 11:01:58.794050 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.793921    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-registry-tls\") pod \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") "
May 29 11:01:58.794050 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.793937    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gbdb\" (UniqueName: \"kubernetes.io/projected/8e26c107-9c4e-4c9e-b370-9fd368371ee5-kube-api-access-7gbdb\") pod \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") "
May 29 11:01:58.794050 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.793967    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-ui-tls\") pod \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\" (UID: \"8e26c107-9c4e-4c9e-b370-9fd368371ee5\") "
May 29 11:01:58.794430 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.794398    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e26c107-9c4e-4c9e-b370-9fd368371ee5-feast-data" (OuterVolumeSpecName: "feast-data") pod "8e26c107-9c4e-4c9e-b370-9fd368371ee5" (UID: "8e26c107-9c4e-4c9e-b370-9fd368371ee5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:01:58.796338 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.796301    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-online-tls" (OuterVolumeSpecName: "online-tls") pod "8e26c107-9c4e-4c9e-b370-9fd368371ee5" (UID: "8e26c107-9c4e-4c9e-b370-9fd368371ee5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:01:58.796338 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.796317    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "8e26c107-9c4e-4c9e-b370-9fd368371ee5" (UID: "8e26c107-9c4e-4c9e-b370-9fd368371ee5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:01:58.796590 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.796565    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "8e26c107-9c4e-4c9e-b370-9fd368371ee5" (UID: "8e26c107-9c4e-4c9e-b370-9fd368371ee5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:01:58.796860 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.796839    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8e26c107-9c4e-4c9e-b370-9fd368371ee5" (UID: "8e26c107-9c4e-4c9e-b370-9fd368371ee5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:01:58.797140 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.797119    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e26c107-9c4e-4c9e-b370-9fd368371ee5-kube-api-access-7gbdb" (OuterVolumeSpecName: "kube-api-access-7gbdb") pod "8e26c107-9c4e-4c9e-b370-9fd368371ee5" (UID: "8e26c107-9c4e-4c9e-b370-9fd368371ee5"). InnerVolumeSpecName "kube-api-access-7gbdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:01:58.895370 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.895283    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.895370 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.895313    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.895370 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.895323    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gbdb\" (UniqueName: \"kubernetes.io/projected/8e26c107-9c4e-4c9e-b370-9fd368371ee5-kube-api-access-7gbdb\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.895370 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.895333    2547 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-ui-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.895370 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.895342    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8e26c107-9c4e-4c9e-b370-9fd368371ee5-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.895370 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.895352    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8e26c107-9c4e-4c9e-b370-9fd368371ee5-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:01:58.961808 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.961293    2547 generic.go:358] "Generic (PLEG): container finished" podID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerID="41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b" exitCode=137
May 29 11:01:58.961808 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.961405    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerDied","Data":"41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b"}
May 29 11:01:58.961808 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.961436    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w" event={"ID":"9ad1f5bf-3db0-454b-8726-f46a778b4c17","Type":"ContainerDied","Data":"d22262a36947e20d1e53fd90d669fc56bd912cec55060ae83bf2738ff1d90028"}
May 29 11:01:58.961808 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.961457    2547 scope.go:117] "RemoveContainer" containerID="cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd"
May 29 11:01:58.961808 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.961626    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"
May 29 11:01:58.965108 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.965075    2547 generic.go:358] "Generic (PLEG): container finished" podID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerID="e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633" exitCode=137
May 29 11:01:58.965108 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.965106    2547 generic.go:358] "Generic (PLEG): container finished" podID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerID="914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e" exitCode=137
May 29 11:01:58.965307 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.965150    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerDied","Data":"e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633"}
May 29 11:01:58.965307 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.965179    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerDied","Data":"914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e"}
May 29 11:01:58.965307 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.965193    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k" event={"ID":"8e26c107-9c4e-4c9e-b370-9fd368371ee5","Type":"ContainerDied","Data":"62cf860ec8ac6e66f410de3694eaa707fd8764e68c09ce63ca90eae6b7981507"}
May 29 11:01:58.965307 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.965215    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"
May 29 11:01:58.971873 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.971848    2547 scope.go:117] "RemoveContainer" containerID="41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b"
May 29 11:01:58.979390 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.979373    2547 scope.go:117] "RemoveContainer" containerID="9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c"
May 29 11:01:58.986208 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.986182    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"]
May 29 11:01:58.986931 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.986915    2547 scope.go:117] "RemoveContainer" containerID="b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9"
May 29 11:01:58.990600 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.990575    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6dfc6b44c4-rhm5w"]
May 29 11:01:58.995010 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:58.994994    2547 scope.go:117] "RemoveContainer" containerID="c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad"
May 29 11:01:59.002838 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.002818    2547 scope.go:117] "RemoveContainer" containerID="cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd"
May 29 11:01:59.003102 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.003084    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd\": container with ID starting with cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd not found: ID does not exist" containerID="cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd"
May 29 11:01:59.003170 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.003107    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd"} err="failed to get container status \"cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd\": rpc error: code = NotFound desc = could not find container \"cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd\": container with ID starting with cf89b9c539174c6703ca623303d081efbea252d52566e5d8c45d6d96b51f62cd not found: ID does not exist"
May 29 11:01:59.003170 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.003125    2547 scope.go:117] "RemoveContainer" containerID="41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b"
May 29 11:01:59.003400 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.003380    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b\": container with ID starting with 41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b not found: ID does not exist" containerID="41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b"
May 29 11:01:59.003442 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.003406    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b"} err="failed to get container status \"41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b\": rpc error: code = NotFound desc = could not find container \"41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b\": container with ID starting with 41fcbc2a34c7f5106239ec5823b0028d61df76b28356dce092734d1c160fcb7b not found: ID does not exist"
May 29 11:01:59.003442 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.003429    2547 scope.go:117] "RemoveContainer" containerID="9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c"
May 29 11:01:59.003521 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.003447    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"]
May 29 11:01:59.003646 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.003632    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c\": container with ID starting with 9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c not found: ID does not exist" containerID="9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c"
May 29 11:01:59.003683 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.003648    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c"} err="failed to get container status \"9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c\": rpc error: code = NotFound desc = could not find container \"9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c\": container with ID starting with 9969e2f10db728d4f4f2ab3843909431caca247ea0dcd53a35df04c12c28b40c not found: ID does not exist"
May 29 11:01:59.003683 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.003660    2547 scope.go:117] "RemoveContainer" containerID="b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9"
May 29 11:01:59.003890 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.003872    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9\": container with ID starting with b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9 not found: ID does not exist" containerID="b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9"
May 29 11:01:59.003940 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.003894    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9"} err="failed to get container status \"b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9\": rpc error: code = NotFound desc = could not find container \"b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9\": container with ID starting with b00240caabaf8bd7454285d6c96548ddbc7c297fe5a8d149aa9a53e4ab5999f9 not found: ID does not exist"
May 29 11:01:59.003940 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.003911    2547 scope.go:117] "RemoveContainer" containerID="c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad"
May 29 11:01:59.004154 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.004135    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad\": container with ID starting with c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad not found: ID does not exist" containerID="c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad"
May 29 11:01:59.004225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.004163    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad"} err="failed to get container status \"c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad\": rpc error: code = NotFound desc = could not find container \"c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad\": container with ID starting with c522decaca3a3d715e32ccd0356c97b32fb2ed1c4858111f1fba1f2e84c58dad not found: ID does not exist"
May 29 11:01:59.004225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.004187    2547 scope.go:117] "RemoveContainer" containerID="390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5"
May 29 11:01:59.008456 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.008432    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-76b886f884-jk67k"]
May 29 11:01:59.012307 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.012286    2547 scope.go:117] "RemoveContainer" containerID="e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633"
May 29 11:01:59.019130 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.019112    2547 scope.go:117] "RemoveContainer" containerID="db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb"
May 29 11:01:59.025707 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.025689    2547 scope.go:117] "RemoveContainer" containerID="914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e"
May 29 11:01:59.032305 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.032289    2547 scope.go:117] "RemoveContainer" containerID="f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d"
May 29 11:01:59.038747 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.038729    2547 scope.go:117] "RemoveContainer" containerID="a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90"
May 29 11:01:59.045349 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.045328    2547 scope.go:117] "RemoveContainer" containerID="390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5"
May 29 11:01:59.045597 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.045578    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5\": container with ID starting with 390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5 not found: ID does not exist" containerID="390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5"
May 29 11:01:59.045653 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.045606    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5"} err="failed to get container status \"390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5\": rpc error: code = NotFound desc = could not find container \"390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5\": container with ID starting with 390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5 not found: ID does not exist"
May 29 11:01:59.045653 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.045625    2547 scope.go:117] "RemoveContainer" containerID="e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633"
May 29 11:01:59.045831 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.045818    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633\": container with ID starting with e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633 not found: ID does not exist" containerID="e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633"
May 29 11:01:59.045874 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.045834    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633"} err="failed to get container status \"e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633\": rpc error: code = NotFound desc = could not find container \"e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633\": container with ID starting with e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633 not found: ID does not exist"
May 29 11:01:59.045874 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.045845    2547 scope.go:117] "RemoveContainer" containerID="db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb"
May 29 11:01:59.046030 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.046012    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb\": container with ID starting with db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb not found: ID does not exist" containerID="db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb"
May 29 11:01:59.046070 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046040    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb"} err="failed to get container status \"db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb\": rpc error: code = NotFound desc = could not find container \"db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb\": container with ID starting with db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb not found: ID does not exist"
May 29 11:01:59.046070 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046057    2547 scope.go:117] "RemoveContainer" containerID="914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e"
May 29 11:01:59.046249 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.046220    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e\": container with ID starting with 914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e not found: ID does not exist" containerID="914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e"
May 29 11:01:59.046299 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046260    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e"} err="failed to get container status \"914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e\": rpc error: code = NotFound desc = could not find container \"914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e\": container with ID starting with 914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e not found: ID does not exist"
May 29 11:01:59.046299 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046277    2547 scope.go:117] "RemoveContainer" containerID="f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d"
May 29 11:01:59.046497 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.046480    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d\": container with ID starting with f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d not found: ID does not exist" containerID="f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d"
May 29 11:01:59.046535 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046501    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d"} err="failed to get container status \"f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d\": rpc error: code = NotFound desc = could not find container \"f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d\": container with ID starting with f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d not found: ID does not exist"
May 29 11:01:59.046535 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046515    2547 scope.go:117] "RemoveContainer" containerID="a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90"
May 29 11:01:59.046729 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:01:59.046716    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90\": container with ID starting with a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90 not found: ID does not exist" containerID="a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90"
May 29 11:01:59.046777 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046733    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90"} err="failed to get container status \"a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90\": rpc error: code = NotFound desc = could not find container \"a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90\": container with ID starting with a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90 not found: ID does not exist"
May 29 11:01:59.046777 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046744    2547 scope.go:117] "RemoveContainer" containerID="390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5"
May 29 11:01:59.046951 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046930    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5"} err="failed to get container status \"390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5\": rpc error: code = NotFound desc = could not find container \"390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5\": container with ID starting with 390076b7b5e45d1a51560a52e3d0b70a11c2bd03c7f42605e7eaa25c13d450c5 not found: ID does not exist"
May 29 11:01:59.046997 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.046952    2547 scope.go:117] "RemoveContainer" containerID="e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633"
May 29 11:01:59.047156 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.047140    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633"} err="failed to get container status \"e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633\": rpc error: code = NotFound desc = could not find container \"e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633\": container with ID starting with e382af7c700cb73527f993bd6a6d45db0d57ccc584e3a795b46648544c9f3633 not found: ID does not exist"
May 29 11:01:59.047195 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.047157    2547 scope.go:117] "RemoveContainer" containerID="db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb"
May 29 11:01:59.047390 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.047369    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb"} err="failed to get container status \"db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb\": rpc error: code = NotFound desc = could not find container \"db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb\": container with ID starting with db0243ba137ebf1cef7b0ae4dd00cec2feca69e649f15c05516e9aceeb5e6abb not found: ID does not exist"
May 29 11:01:59.047443 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.047390    2547 scope.go:117] "RemoveContainer" containerID="914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e"
May 29 11:01:59.047610 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.047592    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e"} err="failed to get container status \"914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e\": rpc error: code = NotFound desc = could not find container \"914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e\": container with ID starting with 914ca4aae33c22b2e75d65dc5ed9394a4f245ede6c0ee1057e8d8df126163d5e not found: ID does not exist"
May 29 11:01:59.047657 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.047611    2547 scope.go:117] "RemoveContainer" containerID="f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d"
May 29 11:01:59.047812 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.047789    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d"} err="failed to get container status \"f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d\": rpc error: code = NotFound desc = could not find container \"f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d\": container with ID starting with f5696db5c38750f0c6a70d7adaba1463cb8e7430a64606e673dabd192ec0a25d not found: ID does not exist"
May 29 11:01:59.047854 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.047814    2547 scope.go:117] "RemoveContainer" containerID="a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90"
May 29 11:01:59.048032 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.048016    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90"} err="failed to get container status \"a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90\": rpc error: code = NotFound desc = could not find container \"a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90\": container with ID starting with a59a3669a97177e0f30443ceb909b86c97414e93fb68fffe0c2295c6707e2e90 not found: ID does not exist"
May 29 11:01:59.717139 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.717104    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" path="/var/lib/kubelet/pods/8e26c107-9c4e-4c9e-b370-9fd368371ee5/volumes"
May 29 11:01:59.717721 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:01:59.717706    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" path="/var/lib/kubelet/pods/9ad1f5bf-3db0-454b-8726-f46a778b4c17/volumes"
May 29 11:02:11.500921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.500847    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2s2bk"]
May 29 11:02:11.501395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501375    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="ui"
May 29 11:02:11.501444 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501400    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="ui"
May 29 11:02:11.501444 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501421    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="online"
May 29 11:02:11.501444 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501430    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="online"
May 29 11:02:11.501444 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501441    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="feast-apply"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501449    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="feast-apply"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501459    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="offline"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501467    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="offline"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501478    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="offline"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501486    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="offline"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501498    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="feast-init"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501507    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="feast-init"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501518    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="ui"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501527    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="ui"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501548    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="feast-apply"
May 29 11:02:11.501573 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501574    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="feast-apply"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501588    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="feast-init"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501598    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="feast-init"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501611    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="registry"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501619    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="registry"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501629    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="online"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501637    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="online"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501715    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="offline"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501725    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="offline"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501736    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="online"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501750    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="ui"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501762    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ad1f5bf-3db0-454b-8726-f46a778b4c17" containerName="online"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501773    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="registry"
May 29 11:02:11.501880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.501782    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e26c107-9c4e-4c9e-b370-9fd368371ee5" containerName="ui"
May 29 11:02:11.505008 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.504986    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:02:11.510053 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.510032    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 29 11:02:11.511727 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.511602    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-jfzzp\""
May 29 11:02:11.511727 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.511653    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2s2bk"]
May 29 11:02:11.511925 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.511847    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 11:02:11.513695 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.513677    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 11:02:11.553566 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.553518    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-854544c579-kzprs"]
May 29 11:02:11.556821 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.556794    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-kzprs"
May 29 11:02:11.566109 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.566086    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-kzprs"]
May 29 11:02:11.602885 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.602849    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/11f3f34b-6ba7-4a73-b8f4-e5552808f995-postgresdata\") pod \"postgres-7c7656cc6d-2s2bk\" (UID: \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\") " pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:02:11.603056 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.602981    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4pnk\" (UniqueName: \"kubernetes.io/projected/11f3f34b-6ba7-4a73-b8f4-e5552808f995-kube-api-access-n4pnk\") pod \"postgres-7c7656cc6d-2s2bk\" (UID: \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\") " pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:02:11.704315 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.704277    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mk7h\" (UniqueName: \"kubernetes.io/projected/04718241-287f-4d65-9adf-881db00861b6-kube-api-access-6mk7h\") pod \"redis-854544c579-kzprs\" (UID: \"04718241-287f-4d65-9adf-881db00861b6\") " pod="test-ns-feast/redis-854544c579-kzprs"
May 29 11:02:11.704485 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.704396    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n4pnk\" (UniqueName: \"kubernetes.io/projected/11f3f34b-6ba7-4a73-b8f4-e5552808f995-kube-api-access-n4pnk\") pod \"postgres-7c7656cc6d-2s2bk\" (UID: \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\") " pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:02:11.704485 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.704457    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/11f3f34b-6ba7-4a73-b8f4-e5552808f995-postgresdata\") pod \"postgres-7c7656cc6d-2s2bk\" (UID: \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\") " pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:02:11.704769 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.704755    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/11f3f34b-6ba7-4a73-b8f4-e5552808f995-postgresdata\") pod \"postgres-7c7656cc6d-2s2bk\" (UID: \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\") " pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:02:11.714285 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.714258    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4pnk\" (UniqueName: \"kubernetes.io/projected/11f3f34b-6ba7-4a73-b8f4-e5552808f995-kube-api-access-n4pnk\") pod \"postgres-7c7656cc6d-2s2bk\" (UID: \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\") " pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:02:11.804823 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.804793    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6mk7h\" (UniqueName: \"kubernetes.io/projected/04718241-287f-4d65-9adf-881db00861b6-kube-api-access-6mk7h\") pod \"redis-854544c579-kzprs\" (UID: \"04718241-287f-4d65-9adf-881db00861b6\") " pod="test-ns-feast/redis-854544c579-kzprs"
May 29 11:02:11.813993 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.813961    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mk7h\" (UniqueName: \"kubernetes.io/projected/04718241-287f-4d65-9adf-881db00861b6-kube-api-access-6mk7h\") pod \"redis-854544c579-kzprs\" (UID: \"04718241-287f-4d65-9adf-881db00861b6\") " pod="test-ns-feast/redis-854544c579-kzprs"
May 29 11:02:11.823094 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.823070    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:02:11.866992 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.866962    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-kzprs"
May 29 11:02:11.962201 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.961825    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2s2bk"]
May 29 11:02:11.964878 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:02:11.964831    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11f3f34b_6ba7_4a73_b8f4_e5552808f995.slice/crio-7b1e92bad5c7de47219f595f3727d3aefa9986d4c69208d8c6fe158bbad5dcc6 WatchSource:0}: Error finding container 7b1e92bad5c7de47219f595f3727d3aefa9986d4c69208d8c6fe158bbad5dcc6: Status 404 returned error can't find the container with id 7b1e92bad5c7de47219f595f3727d3aefa9986d4c69208d8c6fe158bbad5dcc6
May 29 11:02:11.966748 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:11.966730    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 11:02:12.005185 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:12.005157    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-kzprs"]
May 29 11:02:12.007190 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:02:12.007156    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04718241_287f_4d65_9adf_881db00861b6.slice/crio-d300414e58c65bc9844bfe2e333d2d488a55902c0d22379561ad518ff40a3a2c WatchSource:0}: Error finding container d300414e58c65bc9844bfe2e333d2d488a55902c0d22379561ad518ff40a3a2c: Status 404 returned error can't find the container with id d300414e58c65bc9844bfe2e333d2d488a55902c0d22379561ad518ff40a3a2c
May 29 11:02:12.010126 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:12.010104    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk" event={"ID":"11f3f34b-6ba7-4a73-b8f4-e5552808f995","Type":"ContainerStarted","Data":"7b1e92bad5c7de47219f595f3727d3aefa9986d4c69208d8c6fe158bbad5dcc6"}
May 29 11:02:13.014535 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:13.014500    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-kzprs" event={"ID":"04718241-287f-4d65-9adf-881db00861b6","Type":"ContainerStarted","Data":"d300414e58c65bc9844bfe2e333d2d488a55902c0d22379561ad518ff40a3a2c"}
May 29 11:02:14.018550 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:14.018517    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-kzprs" event={"ID":"04718241-287f-4d65-9adf-881db00861b6","Type":"ContainerStarted","Data":"f6ade5dbd5a9842724071d8042a2054172275ea936dfea67baf2a38ec759236f"}
May 29 11:02:14.035560 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:14.035510    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/redis-854544c579-kzprs" podStartSLOduration=1.5643188989999999 podStartE2EDuration="3.035494407s" podCreationTimestamp="2026-05-29 11:02:11 +0000 UTC" firstStartedPulling="2026-05-29 11:02:12.008949652 +0000 UTC m=+710.885511859" lastFinishedPulling="2026-05-29 11:02:13.480125158 +0000 UTC m=+712.356687367" observedRunningTime="2026-05-29 11:02:14.034377861 +0000 UTC m=+712.910940090" watchObservedRunningTime="2026-05-29 11:02:14.035494407 +0000 UTC m=+712.912056635"
May 29 11:02:19.035791 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:19.035753    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk" event={"ID":"11f3f34b-6ba7-4a73-b8f4-e5552808f995","Type":"ContainerStarted","Data":"2c0b37a106485b5ead163cffafbe9eb358466966c8f73e442dfad193e2abc61f"}
May 29 11:02:19.057719 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:19.057675    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk" podStartSLOduration=1.948915726 podStartE2EDuration="8.057663174s" podCreationTimestamp="2026-05-29 11:02:11 +0000 UTC" firstStartedPulling="2026-05-29 11:02:11.966860122 +0000 UTC m=+710.843422329" lastFinishedPulling="2026-05-29 11:02:18.075607561 +0000 UTC m=+716.952169777" observedRunningTime="2026-05-29 11:02:19.055599269 +0000 UTC m=+717.932161497" watchObservedRunningTime="2026-05-29 11:02:19.057663174 +0000 UTC m=+717.934225401"
May 29 11:02:22.353516 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.353472    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"]
May 29 11:02:22.362214 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.362190    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.369998 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.369963    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 29 11:02:22.369998 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.369983    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 29 11:02:22.370193 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.370068    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-tzr8v\""
May 29 11:02:22.370193 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.370178    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 29 11:02:22.370344 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.370328    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 29 11:02:22.379809 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.379781    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"]
May 29 11:02:22.496488 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.496445    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-registry-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.496488 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.496502    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-offline-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.496710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.496529    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csx4l\" (UniqueName: \"kubernetes.io/projected/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-kube-api-access-csx4l\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.496710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.496610    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-feast-data\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.496710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.496648    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-online-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.598039 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.597995    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-feast-data\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.598225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.598055    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-online-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.598225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.598075    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-registry-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.598225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.598096    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-offline-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.598225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.598118    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-csx4l\" (UniqueName: \"kubernetes.io/projected/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-kube-api-access-csx4l\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.598491 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:02:22.598279    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found
May 29 11:02:22.598491 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:02:22.598374    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-online-tls podName:b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d nodeName:}" failed. No retries permitted until 2026-05-29 11:02:23.098347231 +0000 UTC m=+721.974909450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-online-tls") pod "feast-credit-scoring-bd4f8b67c-l5kjd" (UID: "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d") : secret "feast-credit-scoring-online-tls" not found
May 29 11:02:22.598595 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.598490    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-feast-data\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.600656 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.600636    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-registry-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.600721 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.600658    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-offline-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:22.607935 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:22.607874    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csx4l\" (UniqueName: \"kubernetes.io/projected/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-kube-api-access-csx4l\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:23.102076 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:23.102040    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-online-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:23.104500 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:23.104475    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-online-tls\") pod \"feast-credit-scoring-bd4f8b67c-l5kjd\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") " pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:23.271512 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:23.271471    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:23.409813 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:23.409787    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"]
May 29 11:02:23.411749 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:02:23.411717    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dfc5a1_c9a7_47bb_83f4_9c5d42e6973d.slice/crio-b8ba30e9d4cde7199350f864642977b3088f260989f03cf35df18f2c33d2887e WatchSource:0}: Error finding container b8ba30e9d4cde7199350f864642977b3088f260989f03cf35df18f2c33d2887e: Status 404 returned error can't find the container with id b8ba30e9d4cde7199350f864642977b3088f260989f03cf35df18f2c33d2887e
May 29 11:02:24.050200 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:24.050165    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerStarted","Data":"eb493aa23b3172c9899db886a35166c9469a1e8a7df7a98ca58c598ba2a8b17e"}
May 29 11:02:24.050200 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:24.050201    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerStarted","Data":"b8ba30e9d4cde7199350f864642977b3088f260989f03cf35df18f2c33d2887e"}
May 29 11:02:25.054364 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:25.054321    2547 generic.go:358] "Generic (PLEG): container finished" podID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerID="eb493aa23b3172c9899db886a35166c9469a1e8a7df7a98ca58c598ba2a8b17e" exitCode=0
May 29 11:02:25.054770 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:25.054378    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerDied","Data":"eb493aa23b3172c9899db886a35166c9469a1e8a7df7a98ca58c598ba2a8b17e"}
May 29 11:02:26.059131 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:26.059090    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerStarted","Data":"88ff8c4ef8747c9a10545af313774605aca894ce0e12fb377cca461ab63cf9b6"}
May 29 11:02:32.080316 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:32.080284    2547 generic.go:358] "Generic (PLEG): container finished" podID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerID="88ff8c4ef8747c9a10545af313774605aca894ce0e12fb377cca461ab63cf9b6" exitCode=0
May 29 11:02:32.080746 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:32.080343    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerDied","Data":"88ff8c4ef8747c9a10545af313774605aca894ce0e12fb377cca461ab63cf9b6"}
May 29 11:02:33.087103 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:33.087061    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerStarted","Data":"e027493b9e33454d8ecd1fe9aed3d211512734752c446662573d3db2b5b9ded9"}
May 29 11:02:33.087507 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:33.087117    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerStarted","Data":"6556e21b65de9f098573022274a6e4702d6007c64c516bb2fc9c241b480dc73e"}
May 29 11:02:33.087507 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:33.087133    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerStarted","Data":"f3d28d84525d66ddf0671d039dee737794c092a44d402bb6e95b0af20c89ed73"}
May 29 11:02:33.113672 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:33.113607    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podStartSLOduration=11.113585852 podStartE2EDuration="11.113585852s" podCreationTimestamp="2026-05-29 11:02:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:02:33.111303919 +0000 UTC m=+731.987866147" watchObservedRunningTime="2026-05-29 11:02:33.113585852 +0000 UTC m=+731.990148082"
May 29 11:02:33.272214 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:33.272175    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:33.272214 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:33.272224    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:33.272452 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:33.272256    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:35.272652 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:35.272612    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:35.273153 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:35.272729    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:35.273153 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:35.272748    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:35.274597 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:35.274555    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused"
May 29 11:02:35.274597 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:35.274579    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused"
May 29 11:02:35.274780 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:35.274603    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 29 11:02:38.272796 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:38.272749    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused"
May 29 11:02:38.273498 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:38.273473    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:38.274343 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:38.274320    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:38.346487 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:38.346459    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:39.115585 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:39.115548    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:41.271893 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:41.271837    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused"
May 29 11:02:44.272880 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:44.272846    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:02:44.273674 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:02:44.273655    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:03:49.111774 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.111739    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"]
May 29 11:03:49.112309 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.112074    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="registry" containerID="cri-o://f3d28d84525d66ddf0671d039dee737794c092a44d402bb6e95b0af20c89ed73" gracePeriod=30
May 29 11:03:49.112309 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.112148    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="offline" containerID="cri-o://e027493b9e33454d8ecd1fe9aed3d211512734752c446662573d3db2b5b9ded9" gracePeriod=30
May 29 11:03:49.112309 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.112148    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="online" containerID="cri-o://6556e21b65de9f098573022274a6e4702d6007c64c516bb2fc9c241b480dc73e" gracePeriod=30
May 29 11:03:49.130338 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.130306    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2s2bk"]
May 29 11:03:49.130515 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.130495    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk" podUID="11f3f34b-6ba7-4a73-b8f4-e5552808f995" containerName="postgres" containerID="cri-o://2c0b37a106485b5ead163cffafbe9eb358466966c8f73e442dfad193e2abc61f" gracePeriod=30
May 29 11:03:49.139721 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.139690    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-kzprs"]
May 29 11:03:49.140076 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.139941    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/redis-854544c579-kzprs" podUID="04718241-287f-4d65-9adf-881db00861b6" containerName="redis" containerID="cri-o://f6ade5dbd5a9842724071d8042a2054172275ea936dfea67baf2a38ec759236f" gracePeriod=30
May 29 11:03:49.343685 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.343650    2547 generic.go:358] "Generic (PLEG): container finished" podID="04718241-287f-4d65-9adf-881db00861b6" containerID="f6ade5dbd5a9842724071d8042a2054172275ea936dfea67baf2a38ec759236f" exitCode=0
May 29 11:03:49.343839 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.343722    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-kzprs" event={"ID":"04718241-287f-4d65-9adf-881db00861b6","Type":"ContainerDied","Data":"f6ade5dbd5a9842724071d8042a2054172275ea936dfea67baf2a38ec759236f"}
May 29 11:03:49.345320 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.345294    2547 generic.go:358] "Generic (PLEG): container finished" podID="11f3f34b-6ba7-4a73-b8f4-e5552808f995" containerID="2c0b37a106485b5ead163cffafbe9eb358466966c8f73e442dfad193e2abc61f" exitCode=0
May 29 11:03:49.345457 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.345363    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk" event={"ID":"11f3f34b-6ba7-4a73-b8f4-e5552808f995","Type":"ContainerDied","Data":"2c0b37a106485b5ead163cffafbe9eb358466966c8f73e442dfad193e2abc61f"}
May 29 11:03:49.403621 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.403599    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:03:49.406781 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.406756    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-kzprs"
May 29 11:03:49.445794 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.445759    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/11f3f34b-6ba7-4a73-b8f4-e5552808f995-postgresdata\") pod \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\" (UID: \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\") "
May 29 11:03:49.445966 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.445815    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mk7h\" (UniqueName: \"kubernetes.io/projected/04718241-287f-4d65-9adf-881db00861b6-kube-api-access-6mk7h\") pod \"04718241-287f-4d65-9adf-881db00861b6\" (UID: \"04718241-287f-4d65-9adf-881db00861b6\") "
May 29 11:03:49.445966 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.445928    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4pnk\" (UniqueName: \"kubernetes.io/projected/11f3f34b-6ba7-4a73-b8f4-e5552808f995-kube-api-access-n4pnk\") pod \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\" (UID: \"11f3f34b-6ba7-4a73-b8f4-e5552808f995\") "
May 29 11:03:49.448776 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.448723    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f3f34b-6ba7-4a73-b8f4-e5552808f995-kube-api-access-n4pnk" (OuterVolumeSpecName: "kube-api-access-n4pnk") pod "11f3f34b-6ba7-4a73-b8f4-e5552808f995" (UID: "11f3f34b-6ba7-4a73-b8f4-e5552808f995"). InnerVolumeSpecName "kube-api-access-n4pnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:03:49.449586 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.449548    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04718241-287f-4d65-9adf-881db00861b6-kube-api-access-6mk7h" (OuterVolumeSpecName: "kube-api-access-6mk7h") pod "04718241-287f-4d65-9adf-881db00861b6" (UID: "04718241-287f-4d65-9adf-881db00861b6"). InnerVolumeSpecName "kube-api-access-6mk7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:03:49.480948 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.480905    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f3f34b-6ba7-4a73-b8f4-e5552808f995-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "11f3f34b-6ba7-4a73-b8f4-e5552808f995" (UID: "11f3f34b-6ba7-4a73-b8f4-e5552808f995"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:03:49.546691 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.546655    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4pnk\" (UniqueName: \"kubernetes.io/projected/11f3f34b-6ba7-4a73-b8f4-e5552808f995-kube-api-access-n4pnk\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:03:49.546691 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.546687    2547 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/11f3f34b-6ba7-4a73-b8f4-e5552808f995-postgresdata\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:03:49.546691 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:49.546699    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mk7h\" (UniqueName: \"kubernetes.io/projected/04718241-287f-4d65-9adf-881db00861b6-kube-api-access-6mk7h\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:03:50.349317 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.349287    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk"
May 29 11:03:50.349317 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.349297    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-2s2bk" event={"ID":"11f3f34b-6ba7-4a73-b8f4-e5552808f995","Type":"ContainerDied","Data":"7b1e92bad5c7de47219f595f3727d3aefa9986d4c69208d8c6fe158bbad5dcc6"}
May 29 11:03:50.349848 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.349344    2547 scope.go:117] "RemoveContainer" containerID="2c0b37a106485b5ead163cffafbe9eb358466966c8f73e442dfad193e2abc61f"
May 29 11:03:50.352150 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.352127    2547 generic.go:358] "Generic (PLEG): container finished" podID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerID="6556e21b65de9f098573022274a6e4702d6007c64c516bb2fc9c241b480dc73e" exitCode=0
May 29 11:03:50.352255 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.352196    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerDied","Data":"6556e21b65de9f098573022274a6e4702d6007c64c516bb2fc9c241b480dc73e"}
May 29 11:03:50.353362 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.353345    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-kzprs"
May 29 11:03:50.353452 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.353345    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-kzprs" event={"ID":"04718241-287f-4d65-9adf-881db00861b6","Type":"ContainerDied","Data":"d300414e58c65bc9844bfe2e333d2d488a55902c0d22379561ad518ff40a3a2c"}
May 29 11:03:50.357655 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.357639    2547 scope.go:117] "RemoveContainer" containerID="f6ade5dbd5a9842724071d8042a2054172275ea936dfea67baf2a38ec759236f"
May 29 11:03:50.367278 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.367255    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2s2bk"]
May 29 11:03:50.370985 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.370966    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2s2bk"]
May 29 11:03:50.385608 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.385585    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-kzprs"]
May 29 11:03:50.391077 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:50.391051    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-854544c579-kzprs"]
May 29 11:03:51.717498 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:51.717464    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04718241-287f-4d65-9adf-881db00861b6" path="/var/lib/kubelet/pods/04718241-287f-4d65-9adf-881db00861b6/volumes"
May 29 11:03:51.717838 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:51.717800    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f3f34b-6ba7-4a73-b8f4-e5552808f995" path="/var/lib/kubelet/pods/11f3f34b-6ba7-4a73-b8f4-e5552808f995/volumes"
May 29 11:03:59.113208 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:03:59.113120    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 29 11:04:09.113366 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:09.113320    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 29 11:04:19.112822 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.112775    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 29 11:04:19.113222 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.112916    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:04:19.442341 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.442260    2547 generic.go:358] "Generic (PLEG): container finished" podID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerID="e027493b9e33454d8ecd1fe9aed3d211512734752c446662573d3db2b5b9ded9" exitCode=137
May 29 11:04:19.442341 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.442287    2547 generic.go:358] "Generic (PLEG): container finished" podID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerID="f3d28d84525d66ddf0671d039dee737794c092a44d402bb6e95b0af20c89ed73" exitCode=137
May 29 11:04:19.442341 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.442264    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerDied","Data":"e027493b9e33454d8ecd1fe9aed3d211512734752c446662573d3db2b5b9ded9"}
May 29 11:04:19.442341 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.442323    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerDied","Data":"f3d28d84525d66ddf0671d039dee737794c092a44d402bb6e95b0af20c89ed73"}
May 29 11:04:19.751299 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.751274    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:04:19.901790 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.901752    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-registry-tls\") pod \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") "
May 29 11:04:19.901790 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.901798    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-offline-tls\") pod \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") "
May 29 11:04:19.902101 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.901846    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-online-tls\") pod \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") "
May 29 11:04:19.902101 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.901886    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-feast-data\") pod \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") "
May 29 11:04:19.902101 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.901997    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csx4l\" (UniqueName: \"kubernetes.io/projected/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-kube-api-access-csx4l\") pod \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\" (UID: \"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d\") "
May 29 11:04:19.904023 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.903991    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-kube-api-access-csx4l" (OuterVolumeSpecName: "kube-api-access-csx4l") pod "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" (UID: "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d"). InnerVolumeSpecName "kube-api-access-csx4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:04:19.904159 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.904141    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-online-tls" (OuterVolumeSpecName: "online-tls") pod "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" (UID: "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:04:19.904316 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.904291    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" (UID: "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:04:19.904857 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.904830    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" (UID: "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:04:19.911212 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:19.911189    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-feast-data" (OuterVolumeSpecName: "feast-data") pod "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" (UID: "b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:04:20.003131 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.003044    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:04:20.003131 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.003078    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csx4l\" (UniqueName: \"kubernetes.io/projected/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-kube-api-access-csx4l\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:04:20.003131 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.003089    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:04:20.003131 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.003098    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:04:20.003131 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.003106    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:04:20.448021 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.447982    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd" event={"ID":"b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d","Type":"ContainerDied","Data":"b8ba30e9d4cde7199350f864642977b3088f260989f03cf35df18f2c33d2887e"}
May 29 11:04:20.448021 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.448029    2547 scope.go:117] "RemoveContainer" containerID="e027493b9e33454d8ecd1fe9aed3d211512734752c446662573d3db2b5b9ded9"
May 29 11:04:20.448581 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.448067    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"
May 29 11:04:20.456435 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.456415    2547 scope.go:117] "RemoveContainer" containerID="6556e21b65de9f098573022274a6e4702d6007c64c516bb2fc9c241b480dc73e"
May 29 11:04:20.463393 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.463375    2547 scope.go:117] "RemoveContainer" containerID="f3d28d84525d66ddf0671d039dee737794c092a44d402bb6e95b0af20c89ed73"
May 29 11:04:20.470012 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.469989    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"]
May 29 11:04:20.471127 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.471108    2547 scope.go:117] "RemoveContainer" containerID="88ff8c4ef8747c9a10545af313774605aca894ce0e12fb377cca461ab63cf9b6"
May 29 11:04:20.474303 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.474278    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-bd4f8b67c-l5kjd"]
May 29 11:04:20.478795 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:20.478767    2547 scope.go:117] "RemoveContainer" containerID="eb493aa23b3172c9899db886a35166c9469a1e8a7df7a98ca58c598ba2a8b17e"
May 29 11:04:21.718193 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:04:21.718163    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" path="/var/lib/kubelet/pods/b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d/volumes"
May 29 11:05:21.701700 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:05:21.701619    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 11:05:21.702613 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:05:21.702590    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 11:06:16.859404 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859373    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-ld8qm"]
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859699    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="feast-apply"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859710    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="feast-apply"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859725    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="offline"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859733    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="offline"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859748    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04718241-287f-4d65-9adf-881db00861b6" containerName="redis"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859753    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="04718241-287f-4d65-9adf-881db00861b6" containerName="redis"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859763    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="feast-init"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859768    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="feast-init"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859776    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="registry"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859780    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="registry"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859788    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f3f34b-6ba7-4a73-b8f4-e5552808f995" containerName="postgres"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859793    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f3f34b-6ba7-4a73-b8f4-e5552808f995" containerName="postgres"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859803    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="online"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859811    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="online"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859855    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="04718241-287f-4d65-9adf-881db00861b6" containerName="redis"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859865    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="online"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859871    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="registry"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859882    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8dfc5a1-c9a7-47bb-83f4-9c5d42e6973d" containerName="offline"
May 29 11:06:16.861710 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.859891    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="11f3f34b-6ba7-4a73-b8f4-e5552808f995" containerName="postgres"
May 29 11:06:16.862715 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.862698    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-ld8qm"
May 29 11:06:16.866380 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.866353    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 29 11:06:16.867135 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.867107    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-v225m\""
May 29 11:06:16.867454 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.867435    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 29 11:06:16.878372 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.878347    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-ld8qm"]
May 29 11:06:16.953816 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:16.953779    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6gz5\" (UniqueName: \"kubernetes.io/projected/3112eb4f-a44b-475b-99d6-19b888bf40be-kube-api-access-s6gz5\") pod \"redis-854544c579-ld8qm\" (UID: \"3112eb4f-a44b-475b-99d6-19b888bf40be\") " pod="test-ns-feast-rest/redis-854544c579-ld8qm"
May 29 11:06:17.054601 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:17.054568    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s6gz5\" (UniqueName: \"kubernetes.io/projected/3112eb4f-a44b-475b-99d6-19b888bf40be-kube-api-access-s6gz5\") pod \"redis-854544c579-ld8qm\" (UID: \"3112eb4f-a44b-475b-99d6-19b888bf40be\") " pod="test-ns-feast-rest/redis-854544c579-ld8qm"
May 29 11:06:17.075853 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:17.075824    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6gz5\" (UniqueName: \"kubernetes.io/projected/3112eb4f-a44b-475b-99d6-19b888bf40be-kube-api-access-s6gz5\") pod \"redis-854544c579-ld8qm\" (UID: \"3112eb4f-a44b-475b-99d6-19b888bf40be\") " pod="test-ns-feast-rest/redis-854544c579-ld8qm"
May 29 11:06:17.189783 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:17.189695    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-ld8qm"
May 29 11:06:17.312377 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:17.312346    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-ld8qm"]
May 29 11:06:17.315263 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:06:17.315205    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3112eb4f_a44b_475b_99d6_19b888bf40be.slice/crio-ca226efac9b0af597739e87746f0c6ce8e15fde5aa7a1f6f36f7fa00b1d512da WatchSource:0}: Error finding container ca226efac9b0af597739e87746f0c6ce8e15fde5aa7a1f6f36f7fa00b1d512da: Status 404 returned error can't find the container with id ca226efac9b0af597739e87746f0c6ce8e15fde5aa7a1f6f36f7fa00b1d512da
May 29 11:06:17.780859 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:17.780824    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-ld8qm" event={"ID":"3112eb4f-a44b-475b-99d6-19b888bf40be","Type":"ContainerStarted","Data":"97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca"}
May 29 11:06:17.780859 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:17.780858    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-ld8qm" event={"ID":"3112eb4f-a44b-475b-99d6-19b888bf40be","Type":"ContainerStarted","Data":"ca226efac9b0af597739e87746f0c6ce8e15fde5aa7a1f6f36f7fa00b1d512da"}
May 29 11:06:17.796378 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:17.796330    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-ld8qm" podStartSLOduration=1.796314841 podStartE2EDuration="1.796314841s" podCreationTimestamp="2026-05-29 11:06:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:06:17.795520622 +0000 UTC m=+956.672082851" watchObservedRunningTime="2026-05-29 11:06:17.796314841 +0000 UTC m=+956.672877068"
May 29 11:06:22.291294 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.291257    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-24wgb"]
May 29 11:06:22.294557 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.294528    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:06:22.296905 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.296881    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 29 11:06:22.304215 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.304188    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-24wgb"]
May 29 11:06:22.400786 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.400743    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5vmj\" (UniqueName: \"kubernetes.io/projected/2a5af970-9b8e-4018-a974-e81d88649673-kube-api-access-w5vmj\") pod \"postgres-7c7656cc6d-24wgb\" (UID: \"2a5af970-9b8e-4018-a974-e81d88649673\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:06:22.400967 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.400832    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/2a5af970-9b8e-4018-a974-e81d88649673-postgresdata\") pod \"postgres-7c7656cc6d-24wgb\" (UID: \"2a5af970-9b8e-4018-a974-e81d88649673\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:06:22.501474 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.501428    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5vmj\" (UniqueName: \"kubernetes.io/projected/2a5af970-9b8e-4018-a974-e81d88649673-kube-api-access-w5vmj\") pod \"postgres-7c7656cc6d-24wgb\" (UID: \"2a5af970-9b8e-4018-a974-e81d88649673\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:06:22.501647 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.501521    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/2a5af970-9b8e-4018-a974-e81d88649673-postgresdata\") pod \"postgres-7c7656cc6d-24wgb\" (UID: \"2a5af970-9b8e-4018-a974-e81d88649673\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:06:22.501940 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.501920    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/2a5af970-9b8e-4018-a974-e81d88649673-postgresdata\") pod \"postgres-7c7656cc6d-24wgb\" (UID: \"2a5af970-9b8e-4018-a974-e81d88649673\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:06:22.509922 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.509903    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5vmj\" (UniqueName: \"kubernetes.io/projected/2a5af970-9b8e-4018-a974-e81d88649673-kube-api-access-w5vmj\") pod \"postgres-7c7656cc6d-24wgb\" (UID: \"2a5af970-9b8e-4018-a974-e81d88649673\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:06:22.606403 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.606321    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:06:22.731907 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.731879    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-24wgb"]
May 29 11:06:22.734223 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:06:22.734193    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a5af970_9b8e_4018_a974_e81d88649673.slice/crio-0e7ace15a1c83d3db6b8f32411bca74589dd02079afd043eda6b43695d96b85e WatchSource:0}: Error finding container 0e7ace15a1c83d3db6b8f32411bca74589dd02079afd043eda6b43695d96b85e: Status 404 returned error can't find the container with id 0e7ace15a1c83d3db6b8f32411bca74589dd02079afd043eda6b43695d96b85e
May 29 11:06:22.796678 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.796647    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb" event={"ID":"2a5af970-9b8e-4018-a974-e81d88649673","Type":"ContainerStarted","Data":"c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469"}
May 29 11:06:22.796883 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.796683    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb" event={"ID":"2a5af970-9b8e-4018-a974-e81d88649673","Type":"ContainerStarted","Data":"0e7ace15a1c83d3db6b8f32411bca74589dd02079afd043eda6b43695d96b85e"}
May 29 11:06:22.814299 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:22.814205    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb" podStartSLOduration=0.814190776 podStartE2EDuration="814.190776ms" podCreationTimestamp="2026-05-29 11:06:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:06:22.812795199 +0000 UTC m=+961.689357427" watchObservedRunningTime="2026-05-29 11:06:22.814190776 +0000 UTC m=+961.690753004"
May 29 11:06:27.965360 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:27.965325    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"]
May 29 11:06:27.968885 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:27.968864    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:27.971249 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:27.971216    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 29 11:06:27.971350 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:27.971218    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 29 11:06:27.971464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:27.971444    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 29 11:06:27.971511 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:27.971444    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-n5qrh\""
May 29 11:06:27.980018 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:27.979998    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"]
May 29 11:06:28.044467 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.044432    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zptb\" (UniqueName: \"kubernetes.io/projected/88598156-6c72-42b2-842e-88096c72d9dd-kube-api-access-9zptb\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.044629 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.044488    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-registry-tls\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.044629 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.044547    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88598156-6c72-42b2-842e-88096c72d9dd-feast-data\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.044629 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.044610    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-online-tls\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.145052 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.145005    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88598156-6c72-42b2-842e-88096c72d9dd-feast-data\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.145052 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.145055    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-online-tls\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.145294 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.145111    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9zptb\" (UniqueName: \"kubernetes.io/projected/88598156-6c72-42b2-842e-88096c72d9dd-kube-api-access-9zptb\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.145294 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.145146    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-registry-tls\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.145294 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:06:28.145232    2547 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found
May 29 11:06:28.145392 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:06:28.145325    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-online-tls podName:88598156-6c72-42b2-842e-88096c72d9dd nodeName:}" failed. No retries permitted until 2026-05-29 11:06:28.645303385 +0000 UTC m=+967.521865605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-online-tls") pod "feast-credit-scoring-69b65dcc6c-nvddv" (UID: "88598156-6c72-42b2-842e-88096c72d9dd") : secret "feast-credit-scoring-online-tls" not found
May 29 11:06:28.145454 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.145431    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88598156-6c72-42b2-842e-88096c72d9dd-feast-data\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.147706 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.147678    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-registry-tls\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.153345 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.153322    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zptb\" (UniqueName: \"kubernetes.io/projected/88598156-6c72-42b2-842e-88096c72d9dd-kube-api-access-9zptb\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.650617 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.650576    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-online-tls\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.652953 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.652925    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-online-tls\") pod \"feast-credit-scoring-69b65dcc6c-nvddv\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") " pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:28.878313 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:28.878281    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:29.007420 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:29.007388    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"]
May 29 11:06:29.011379 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:06:29.011340    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88598156_6c72_42b2_842e_88096c72d9dd.slice/crio-20fbf6753e99f9ff4870236c080f174caf3bcdec1a5ea56a271780ad506ca92c WatchSource:0}: Error finding container 20fbf6753e99f9ff4870236c080f174caf3bcdec1a5ea56a271780ad506ca92c: Status 404 returned error can't find the container with id 20fbf6753e99f9ff4870236c080f174caf3bcdec1a5ea56a271780ad506ca92c
May 29 11:06:29.821463 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:29.821418    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerStarted","Data":"db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21"}
May 29 11:06:29.821463 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:29.821465    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerStarted","Data":"20fbf6753e99f9ff4870236c080f174caf3bcdec1a5ea56a271780ad506ca92c"}
May 29 11:06:30.825227 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:30.825193    2547 generic.go:358] "Generic (PLEG): container finished" podID="88598156-6c72-42b2-842e-88096c72d9dd" containerID="db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21" exitCode=0
May 29 11:06:30.825660 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:30.825273    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerDied","Data":"db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21"}
May 29 11:06:31.830177 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:31.830137    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerStarted","Data":"ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05"}
May 29 11:06:38.860150 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:38.860117    2547 generic.go:358] "Generic (PLEG): container finished" podID="88598156-6c72-42b2-842e-88096c72d9dd" containerID="ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05" exitCode=0
May 29 11:06:38.860544 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:38.860167    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerDied","Data":"ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05"}
May 29 11:06:39.866115 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:39.866078    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerStarted","Data":"927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7"}
May 29 11:06:39.866115 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:39.866123    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerStarted","Data":"d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56"}
May 29 11:06:39.889311 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:39.889224    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podStartSLOduration=12.88920519 podStartE2EDuration="12.88920519s" podCreationTimestamp="2026-05-29 11:06:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:06:39.887591625 +0000 UTC m=+978.764153866" watchObservedRunningTime="2026-05-29 11:06:39.88920519 +0000 UTC m=+978.765767418"
May 29 11:06:40.879304 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:40.879272    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:40.879802 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:40.879318    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:40.880786 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:40.880754    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 29 11:06:40.880890 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:40.880781    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 29 11:06:43.879481 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:43.879429    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 29 11:06:43.879481 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:43.879454    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 29 11:06:46.879525 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:46.879475    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 29 11:06:46.884768 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:46.884739    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:46.885301 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:46.885278    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:46.888094 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:46.888073    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:48.879069 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:48.879025    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:49.880025 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:49.879996    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:49.899399 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:49.899371    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:06:53.806197 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.806158    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"]
May 29 11:06:53.811102 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.811084    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.813197 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.813172    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 29 11:06:53.813330 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.813295    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 29 11:06:53.813330 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.813314    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 29 11:06:53.813420 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.813374    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-ddcvt\""
May 29 11:06:53.819437 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.819413    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"]
May 29 11:06:53.879471 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.879443    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x6jw\" (UniqueName: \"kubernetes.io/projected/aff494c0-2142-49d1-aa04-b2574cc5e6c0-kube-api-access-5x6jw\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.879624 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.879477    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-registry-tls\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.879624 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.879559    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-online-tls\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.879624 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.879597    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aff494c0-2142-49d1-aa04-b2574cc5e6c0-feast-data\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.980651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.980620    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5x6jw\" (UniqueName: \"kubernetes.io/projected/aff494c0-2142-49d1-aa04-b2574cc5e6c0-kube-api-access-5x6jw\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.980651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.980652    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-registry-tls\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.980870 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.980705    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-online-tls\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.980870 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.980725    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aff494c0-2142-49d1-aa04-b2574cc5e6c0-feast-data\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.980870 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:06:53.980842    2547 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found
May 29 11:06:53.980986 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:06:53.980916    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-online-tls podName:aff494c0-2142-49d1-aa04-b2574cc5e6c0 nodeName:}" failed. No retries permitted until 2026-05-29 11:06:54.480894904 +0000 UTC m=+993.357457111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-online-tls") pod "feast-driver-ranking-6cd86c886d-64d2b" (UID: "aff494c0-2142-49d1-aa04-b2574cc5e6c0") : secret "feast-driver-ranking-online-tls" not found
May 29 11:06:53.981162 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.981143    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aff494c0-2142-49d1-aa04-b2574cc5e6c0-feast-data\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.983144 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.983127    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-registry-tls\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:53.996635 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:53.996613    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x6jw\" (UniqueName: \"kubernetes.io/projected/aff494c0-2142-49d1-aa04-b2574cc5e6c0-kube-api-access-5x6jw\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:54.484342 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:54.484298    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-online-tls\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:54.486837 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:54.486810    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-online-tls\") pod \"feast-driver-ranking-6cd86c886d-64d2b\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") " pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:54.721197 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:54.721156    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:06:54.856844 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:54.856743    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"]
May 29 11:06:54.859173 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:06:54.859139    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff494c0_2142_49d1_aa04_b2574cc5e6c0.slice/crio-6ccf22ecc73e2294c4db54ade9dc1f607569d63ea68e96fd0bb072457e4eed8a WatchSource:0}: Error finding container 6ccf22ecc73e2294c4db54ade9dc1f607569d63ea68e96fd0bb072457e4eed8a: Status 404 returned error can't find the container with id 6ccf22ecc73e2294c4db54ade9dc1f607569d63ea68e96fd0bb072457e4eed8a
May 29 11:06:54.916489 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:54.916459    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerStarted","Data":"6ccf22ecc73e2294c4db54ade9dc1f607569d63ea68e96fd0bb072457e4eed8a"}
May 29 11:06:55.920519 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:55.920484    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerStarted","Data":"42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465"}
May 29 11:06:59.934194 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:59.934158    2547 generic.go:358] "Generic (PLEG): container finished" podID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerID="42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465" exitCode=0
May 29 11:06:59.934593 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:06:59.934233    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerDied","Data":"42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465"}
May 29 11:07:00.939315 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:00.939281    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerStarted","Data":"bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253"}
May 29 11:07:06.959529 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:06.959494    2547 generic.go:358] "Generic (PLEG): container finished" podID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerID="bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253" exitCode=0
May 29 11:07:06.959857 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:06.959570    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerDied","Data":"bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253"}
May 29 11:07:07.964917 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:07.964876    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerStarted","Data":"103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4"}
May 29 11:07:07.965526 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:07.964929    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerStarted","Data":"ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab"}
May 29 11:07:07.987824 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:07.987591    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podStartSLOduration=14.987574082 podStartE2EDuration="14.987574082s" podCreationTimestamp="2026-05-29 11:06:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:07:07.985945175 +0000 UTC m=+1006.862507426" watchObservedRunningTime="2026-05-29 11:07:07.987574082 +0000 UTC m=+1006.864136310"
May 29 11:07:09.722143 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:09.722111    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:07:09.722143 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:09.722154    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:07:09.723789 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:09.723758    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused"
May 29 11:07:09.723908 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:09.723794    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 29 11:07:12.721946 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:12.721900    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused"
May 29 11:07:13.097747 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:13.097710    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:07:13.098223 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:13.098195    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:07:13.101105 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:13.101082    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:07:14.722162 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:14.722120    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:07:15.721621 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:15.721573    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused"
May 29 11:07:18.722709 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:18.722678    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:07:18.723317 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:07:18.723299    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:08:34.044412 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.044374    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"]
May 29 11:08:34.044888 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.044611    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg" podUID="e7b90bb3-5012-4d14-9986-7a44ee5e2141" containerName="manager" containerID="cri-o://bb1190d9a5850f0fa80ae84351d7cb64018d42edf80408af0dd1e70b36fdfc0b" gracePeriod=10
May 29 11:08:34.238201 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.238161    2547 generic.go:358] "Generic (PLEG): container finished" podID="e7b90bb3-5012-4d14-9986-7a44ee5e2141" containerID="bb1190d9a5850f0fa80ae84351d7cb64018d42edf80408af0dd1e70b36fdfc0b" exitCode=0
May 29 11:08:34.238386 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.238326    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg" event={"ID":"e7b90bb3-5012-4d14-9986-7a44ee5e2141","Type":"ContainerDied","Data":"bb1190d9a5850f0fa80ae84351d7cb64018d42edf80408af0dd1e70b36fdfc0b"}
May 29 11:08:34.325171 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.324805    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"]
May 29 11:08:34.325492 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.325189    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="registry" containerID="cri-o://d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56" gracePeriod=30
May 29 11:08:34.325637 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.325536    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="online" containerID="cri-o://927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7" gracePeriod=30
May 29 11:08:34.331610 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.331584    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"]
May 29 11:08:34.332005 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.331965    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="registry" containerID="cri-o://ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab" gracePeriod=30
May 29 11:08:34.332110 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.332017    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="online" containerID="cri-o://103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4" gracePeriod=30
May 29 11:08:34.339839 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.339820    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"
May 29 11:08:34.458868 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.458829    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl66s\" (UniqueName: \"kubernetes.io/projected/e7b90bb3-5012-4d14-9986-7a44ee5e2141-kube-api-access-fl66s\") pod \"e7b90bb3-5012-4d14-9986-7a44ee5e2141\" (UID: \"e7b90bb3-5012-4d14-9986-7a44ee5e2141\") "
May 29 11:08:34.460981 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.460952    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b90bb3-5012-4d14-9986-7a44ee5e2141-kube-api-access-fl66s" (OuterVolumeSpecName: "kube-api-access-fl66s") pod "e7b90bb3-5012-4d14-9986-7a44ee5e2141" (UID: "e7b90bb3-5012-4d14-9986-7a44ee5e2141"). InnerVolumeSpecName "kube-api-access-fl66s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:08:34.559980 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:34.559941    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl66s\" (UniqueName: \"kubernetes.io/projected/e7b90bb3-5012-4d14-9986-7a44ee5e2141-kube-api-access-fl66s\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:08:35.243130 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.243094    2547 generic.go:358] "Generic (PLEG): container finished" podID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerID="103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4" exitCode=0
May 29 11:08:35.243532 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.243171    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerDied","Data":"103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4"}
May 29 11:08:35.244983 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.244964    2547 generic.go:358] "Generic (PLEG): container finished" podID="88598156-6c72-42b2-842e-88096c72d9dd" containerID="927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7" exitCode=0
May 29 11:08:35.245051 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.245032    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerDied","Data":"927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7"}
May 29 11:08:35.245961 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.245940    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg" event={"ID":"e7b90bb3-5012-4d14-9986-7a44ee5e2141","Type":"ContainerDied","Data":"4215cbd61b5c7bcf94e2ff15ac5a1561312ecbbf60862ccb9baf197ea4b7cb0f"}
May 29 11:08:35.246072 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.245971    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"
May 29 11:08:35.246072 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.245973    2547 scope.go:117] "RemoveContainer" containerID="bb1190d9a5850f0fa80ae84351d7cb64018d42edf80408af0dd1e70b36fdfc0b"
May 29 11:08:35.266171 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.266139    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"]
May 29 11:08:35.269443 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.269421    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6fccb7dc4-zf6lg"]
May 29 11:08:35.386501 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.386419    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-24wgb"]
May 29 11:08:35.386639 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.386621    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb" podUID="2a5af970-9b8e-4018-a974-e81d88649673" containerName="postgres" containerID="cri-o://c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469" gracePeriod=30
May 29 11:08:35.391345 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.391319    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-ld8qm"]
May 29 11:08:35.391557 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.391536    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-ld8qm" podUID="3112eb4f-a44b-475b-99d6-19b888bf40be" containerName="redis" containerID="cri-o://97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca" gracePeriod=30
May 29 11:08:35.646227 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.646207    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:08:35.649412 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.649393    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-ld8qm"
May 29 11:08:35.718823 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.718776    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b90bb3-5012-4d14-9986-7a44ee5e2141" path="/var/lib/kubelet/pods/e7b90bb3-5012-4d14-9986-7a44ee5e2141/volumes"
May 29 11:08:35.771740 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.771709    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6gz5\" (UniqueName: \"kubernetes.io/projected/3112eb4f-a44b-475b-99d6-19b888bf40be-kube-api-access-s6gz5\") pod \"3112eb4f-a44b-475b-99d6-19b888bf40be\" (UID: \"3112eb4f-a44b-475b-99d6-19b888bf40be\") "
May 29 11:08:35.771919 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.771781    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5vmj\" (UniqueName: \"kubernetes.io/projected/2a5af970-9b8e-4018-a974-e81d88649673-kube-api-access-w5vmj\") pod \"2a5af970-9b8e-4018-a974-e81d88649673\" (UID: \"2a5af970-9b8e-4018-a974-e81d88649673\") "
May 29 11:08:35.771919 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.771805    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/2a5af970-9b8e-4018-a974-e81d88649673-postgresdata\") pod \"2a5af970-9b8e-4018-a974-e81d88649673\" (UID: \"2a5af970-9b8e-4018-a974-e81d88649673\") "
May 29 11:08:35.773884 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.773849    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3112eb4f-a44b-475b-99d6-19b888bf40be-kube-api-access-s6gz5" (OuterVolumeSpecName: "kube-api-access-s6gz5") pod "3112eb4f-a44b-475b-99d6-19b888bf40be" (UID: "3112eb4f-a44b-475b-99d6-19b888bf40be"). InnerVolumeSpecName "kube-api-access-s6gz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:08:35.773983 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.773914    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5af970-9b8e-4018-a974-e81d88649673-kube-api-access-w5vmj" (OuterVolumeSpecName: "kube-api-access-w5vmj") pod "2a5af970-9b8e-4018-a974-e81d88649673" (UID: "2a5af970-9b8e-4018-a974-e81d88649673"). InnerVolumeSpecName "kube-api-access-w5vmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:08:35.802009 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.801965    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a5af970-9b8e-4018-a974-e81d88649673-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "2a5af970-9b8e-4018-a974-e81d88649673" (UID: "2a5af970-9b8e-4018-a974-e81d88649673"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:08:35.872554 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.872520    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6gz5\" (UniqueName: \"kubernetes.io/projected/3112eb4f-a44b-475b-99d6-19b888bf40be-kube-api-access-s6gz5\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:08:35.872554 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.872550    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5vmj\" (UniqueName: \"kubernetes.io/projected/2a5af970-9b8e-4018-a974-e81d88649673-kube-api-access-w5vmj\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:08:35.872734 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:35.872560    2547 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/2a5af970-9b8e-4018-a974-e81d88649673-postgresdata\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:08:36.250809 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.250768    2547 generic.go:358] "Generic (PLEG): container finished" podID="2a5af970-9b8e-4018-a974-e81d88649673" containerID="c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469" exitCode=0
May 29 11:08:36.251252 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.250842    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb"
May 29 11:08:36.251252 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.250851    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb" event={"ID":"2a5af970-9b8e-4018-a974-e81d88649673","Type":"ContainerDied","Data":"c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469"}
May 29 11:08:36.251252 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.250889    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-24wgb" event={"ID":"2a5af970-9b8e-4018-a974-e81d88649673","Type":"ContainerDied","Data":"0e7ace15a1c83d3db6b8f32411bca74589dd02079afd043eda6b43695d96b85e"}
May 29 11:08:36.251252 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.250904    2547 scope.go:117] "RemoveContainer" containerID="c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469"
May 29 11:08:36.252111 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.252064    2547 generic.go:358] "Generic (PLEG): container finished" podID="3112eb4f-a44b-475b-99d6-19b888bf40be" containerID="97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca" exitCode=0
May 29 11:08:36.252111 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.252096    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-ld8qm" event={"ID":"3112eb4f-a44b-475b-99d6-19b888bf40be","Type":"ContainerDied","Data":"97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca"}
May 29 11:08:36.252214 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.252127    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-ld8qm"
May 29 11:08:36.252214 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.252129    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-ld8qm" event={"ID":"3112eb4f-a44b-475b-99d6-19b888bf40be","Type":"ContainerDied","Data":"ca226efac9b0af597739e87746f0c6ce8e15fde5aa7a1f6f36f7fa00b1d512da"}
May 29 11:08:36.258749 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.258730    2547 scope.go:117] "RemoveContainer" containerID="c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469"
May 29 11:08:36.259017 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:08:36.258993    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469\": container with ID starting with c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469 not found: ID does not exist" containerID="c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469"
May 29 11:08:36.259112 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.259021    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469"} err="failed to get container status \"c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469\": rpc error: code = NotFound desc = could not find container \"c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469\": container with ID starting with c10c14e96d51f1de9c5d8a321842d5ddadd69941e8c57678cbf41ef1ecfd5469 not found: ID does not exist"
May 29 11:08:36.259112 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.259039    2547 scope.go:117] "RemoveContainer" containerID="97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca"
May 29 11:08:36.266874 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.266858    2547 scope.go:117] "RemoveContainer" containerID="97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca"
May 29 11:08:36.267110 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:08:36.267093    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca\": container with ID starting with 97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca not found: ID does not exist" containerID="97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca"
May 29 11:08:36.267171 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.267121    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca"} err="failed to get container status \"97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca\": rpc error: code = NotFound desc = could not find container \"97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca\": container with ID starting with 97d98d0a8ee6122e6691579332edd2cd8f44d6e94d48ff3f015fd89d47aaeaca not found: ID does not exist"
May 29 11:08:36.274856 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.274834    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-24wgb"]
May 29 11:08:36.278227 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.278209    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-24wgb"]
May 29 11:08:36.288084 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.288065    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-ld8qm"]
May 29 11:08:36.291944 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.291924    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-ld8qm"]
May 29 11:08:36.886196 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:36.886158    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 29 11:08:37.717057 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:37.717023    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5af970-9b8e-4018-a974-e81d88649673" path="/var/lib/kubelet/pods/2a5af970-9b8e-4018-a974-e81d88649673/volumes"
May 29 11:08:37.717438 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:37.717347    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3112eb4f-a44b-475b-99d6-19b888bf40be" path="/var/lib/kubelet/pods/3112eb4f-a44b-475b-99d6-19b888bf40be/volumes"
May 29 11:08:43.098619 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:43.098573    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 29 11:08:45.841057 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841027    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"]
May 29 11:08:45.841462 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841359    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5af970-9b8e-4018-a974-e81d88649673" containerName="postgres"
May 29 11:08:45.841462 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841372    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5af970-9b8e-4018-a974-e81d88649673" containerName="postgres"
May 29 11:08:45.841462 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841398    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b90bb3-5012-4d14-9986-7a44ee5e2141" containerName="manager"
May 29 11:08:45.841462 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841403    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b90bb3-5012-4d14-9986-7a44ee5e2141" containerName="manager"
May 29 11:08:45.841462 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841410    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3112eb4f-a44b-475b-99d6-19b888bf40be" containerName="redis"
May 29 11:08:45.841462 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841415    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="3112eb4f-a44b-475b-99d6-19b888bf40be" containerName="redis"
May 29 11:08:45.841462 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841464    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="e7b90bb3-5012-4d14-9986-7a44ee5e2141" containerName="manager"
May 29 11:08:45.841725 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841473    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a5af970-9b8e-4018-a974-e81d88649673" containerName="postgres"
May 29 11:08:45.841725 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.841481    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="3112eb4f-a44b-475b-99d6-19b888bf40be" containerName="redis"
May 29 11:08:45.844430 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.844409    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"
May 29 11:08:45.847064 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.847044    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 29 11:08:45.847661 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.847641    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-zh8ts\""
May 29 11:08:45.848265 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.848234    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 29 11:08:45.866070 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.866037    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"]
May 29 11:08:45.957422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:45.957389    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7blp\" (UniqueName: \"kubernetes.io/projected/dc4a531e-48d9-4811-8344-e176e5bcdd65-kube-api-access-d7blp\") pod \"feast-operator-controller-manager-cf99db75-4794v\" (UID: \"dc4a531e-48d9-4811-8344-e176e5bcdd65\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"
May 29 11:08:46.058540 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:46.058505    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d7blp\" (UniqueName: \"kubernetes.io/projected/dc4a531e-48d9-4811-8344-e176e5bcdd65-kube-api-access-d7blp\") pod \"feast-operator-controller-manager-cf99db75-4794v\" (UID: \"dc4a531e-48d9-4811-8344-e176e5bcdd65\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"
May 29 11:08:46.068763 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:46.068733    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7blp\" (UniqueName: \"kubernetes.io/projected/dc4a531e-48d9-4811-8344-e176e5bcdd65-kube-api-access-d7blp\") pod \"feast-operator-controller-manager-cf99db75-4794v\" (UID: \"dc4a531e-48d9-4811-8344-e176e5bcdd65\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"
May 29 11:08:46.154478 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:46.154393    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"
May 29 11:08:46.282477 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:46.282407    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"]
May 29 11:08:46.285472 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:08:46.285446    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4a531e_48d9_4811_8344_e176e5bcdd65.slice/crio-ce96e4d0d933893161c95b6be0ffa7160c3ed5cc397a494b80ae67c4f2c2ae17 WatchSource:0}: Error finding container ce96e4d0d933893161c95b6be0ffa7160c3ed5cc397a494b80ae67c4f2c2ae17: Status 404 returned error can't find the container with id ce96e4d0d933893161c95b6be0ffa7160c3ed5cc397a494b80ae67c4f2c2ae17
May 29 11:08:46.287063 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:46.287048    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 11:08:46.885937 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:46.885901    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 29 11:08:47.289071 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:47.289034    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v" event={"ID":"dc4a531e-48d9-4811-8344-e176e5bcdd65","Type":"ContainerStarted","Data":"ce96e4d0d933893161c95b6be0ffa7160c3ed5cc397a494b80ae67c4f2c2ae17"}
May 29 11:08:49.297772 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:49.297730    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v" event={"ID":"dc4a531e-48d9-4811-8344-e176e5bcdd65","Type":"ContainerStarted","Data":"b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b"}
May 29 11:08:49.298129 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:49.297882    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"
May 29 11:08:49.318166 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:49.318116    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v" podStartSLOduration=2.178557333 podStartE2EDuration="4.318102055s" podCreationTimestamp="2026-05-29 11:08:45 +0000 UTC" firstStartedPulling="2026-05-29 11:08:46.287168407 +0000 UTC m=+1105.163730613" lastFinishedPulling="2026-05-29 11:08:48.426713125 +0000 UTC m=+1107.303275335" observedRunningTime="2026-05-29 11:08:49.316087099 +0000 UTC m=+1108.192649326" watchObservedRunningTime="2026-05-29 11:08:49.318102055 +0000 UTC m=+1108.194664298"
May 29 11:08:53.098769 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:53.098728    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 29 11:08:56.886129 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:56.886092    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 29 11:08:56.886540 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:08:56.886199    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:09:00.303652 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:00.303621    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"
May 29 11:09:03.099317 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:03.099266    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 29 11:09:03.099689 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:03.099429    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:09:04.993777 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:04.993751    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:09:04.996764 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:04.996749    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:09:05.022859 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.022836    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-registry-tls\") pod \"88598156-6c72-42b2-842e-88096c72d9dd\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") "
May 29 11:09:05.023125 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.022872    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x6jw\" (UniqueName: \"kubernetes.io/projected/aff494c0-2142-49d1-aa04-b2574cc5e6c0-kube-api-access-5x6jw\") pod \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") "
May 29 11:09:05.023125 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.022904    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-online-tls\") pod \"88598156-6c72-42b2-842e-88096c72d9dd\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") "
May 29 11:09:05.023125 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.022932    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-online-tls\") pod \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") "
May 29 11:09:05.023125 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.022959    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zptb\" (UniqueName: \"kubernetes.io/projected/88598156-6c72-42b2-842e-88096c72d9dd-kube-api-access-9zptb\") pod \"88598156-6c72-42b2-842e-88096c72d9dd\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") "
May 29 11:09:05.023125 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.023008    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88598156-6c72-42b2-842e-88096c72d9dd-feast-data\") pod \"88598156-6c72-42b2-842e-88096c72d9dd\" (UID: \"88598156-6c72-42b2-842e-88096c72d9dd\") "
May 29 11:09:05.023125 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.023029    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-registry-tls\") pod \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") "
May 29 11:09:05.023125 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.023056    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aff494c0-2142-49d1-aa04-b2574cc5e6c0-feast-data\") pod \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\" (UID: \"aff494c0-2142-49d1-aa04-b2574cc5e6c0\") "
May 29 11:09:05.024042 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.023981    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff494c0-2142-49d1-aa04-b2574cc5e6c0-feast-data" (OuterVolumeSpecName: "feast-data") pod "aff494c0-2142-49d1-aa04-b2574cc5e6c0" (UID: "aff494c0-2142-49d1-aa04-b2574cc5e6c0"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:09:05.025624 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.025596    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "88598156-6c72-42b2-842e-88096c72d9dd" (UID: "88598156-6c72-42b2-842e-88096c72d9dd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:09:05.025728 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.025625    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-online-tls" (OuterVolumeSpecName: "online-tls") pod "88598156-6c72-42b2-842e-88096c72d9dd" (UID: "88598156-6c72-42b2-842e-88096c72d9dd"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:09:05.025787 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.025750    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88598156-6c72-42b2-842e-88096c72d9dd-kube-api-access-9zptb" (OuterVolumeSpecName: "kube-api-access-9zptb") pod "88598156-6c72-42b2-842e-88096c72d9dd" (UID: "88598156-6c72-42b2-842e-88096c72d9dd"). InnerVolumeSpecName "kube-api-access-9zptb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:09:05.025787 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.025754    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-online-tls" (OuterVolumeSpecName: "online-tls") pod "aff494c0-2142-49d1-aa04-b2574cc5e6c0" (UID: "aff494c0-2142-49d1-aa04-b2574cc5e6c0"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:09:05.025923 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.025889    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "aff494c0-2142-49d1-aa04-b2574cc5e6c0" (UID: "aff494c0-2142-49d1-aa04-b2574cc5e6c0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:09:05.026671 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.026648    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff494c0-2142-49d1-aa04-b2574cc5e6c0-kube-api-access-5x6jw" (OuterVolumeSpecName: "kube-api-access-5x6jw") pod "aff494c0-2142-49d1-aa04-b2574cc5e6c0" (UID: "aff494c0-2142-49d1-aa04-b2574cc5e6c0"). InnerVolumeSpecName "kube-api-access-5x6jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:09:05.037494 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.037448    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88598156-6c72-42b2-842e-88096c72d9dd-feast-data" (OuterVolumeSpecName: "feast-data") pod "88598156-6c72-42b2-842e-88096c72d9dd" (UID: "88598156-6c72-42b2-842e-88096c72d9dd"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:09:05.123980 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.123904    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/88598156-6c72-42b2-842e-88096c72d9dd-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:09:05.123980 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.123939    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:09:05.123980 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.123953    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aff494c0-2142-49d1-aa04-b2574cc5e6c0-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:09:05.123980 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.123966    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:09:05.123980 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.123980    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x6jw\" (UniqueName: \"kubernetes.io/projected/aff494c0-2142-49d1-aa04-b2574cc5e6c0-kube-api-access-5x6jw\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:09:05.124258 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.123996    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/88598156-6c72-42b2-842e-88096c72d9dd-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:09:05.124258 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.124010    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aff494c0-2142-49d1-aa04-b2574cc5e6c0-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:09:05.124258 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.124023    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zptb\" (UniqueName: \"kubernetes.io/projected/88598156-6c72-42b2-842e-88096c72d9dd-kube-api-access-9zptb\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:09:05.289028 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.288986    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"]
May 29 11:09:05.289488 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289463    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="registry"
May 29 11:09:05.289488 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289483    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="registry"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289499    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="online"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289508    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="online"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289522    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="feast-init"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289530    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="feast-init"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289545    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="registry"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289553    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="registry"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289569    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="feast-apply"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289577    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="feast-apply"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289586    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="feast-init"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289594    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="feast-init"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289606    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="online"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289614    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="online"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289624    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="feast-apply"
May 29 11:09:05.289651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289632    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="feast-apply"
May 29 11:09:05.290089 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289700    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="registry"
May 29 11:09:05.290089 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289714    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="88598156-6c72-42b2-842e-88096c72d9dd" containerName="online"
May 29 11:09:05.290089 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289724    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="registry"
May 29 11:09:05.290089 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.289734    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerName="online"
May 29 11:09:05.294825 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.294806    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.297399 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.297376    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 29 11:09:05.297399 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.297392    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 11:09:05.297601 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.297393    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 11:09:05.297689 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.297670    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 29 11:09:05.297752 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.297685    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 29 11:09:05.297752 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.297700    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 29 11:09:05.297856 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.297786    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-t8nds\""
May 29 11:09:05.303384 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.303360    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"]
May 29 11:09:05.325709 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.325684    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.325844 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.325723    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.325844 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.325742    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcad2ea-42a2-4e81-ad33-49abaee247e0-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.325844 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.325825    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-958sg\" (UniqueName: \"kubernetes.io/projected/fbcad2ea-42a2-4e81-ad33-49abaee247e0-kube-api-access-958sg\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.325979 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.325874    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.325979 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.325911    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.349364 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.349334    2547 generic.go:358] "Generic (PLEG): container finished" podID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" containerID="ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab" exitCode=137
May 29 11:09:05.349492 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.349405    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"
May 29 11:09:05.349492 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.349412    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerDied","Data":"ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab"}
May 29 11:09:05.349492 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.349444    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b" event={"ID":"aff494c0-2142-49d1-aa04-b2574cc5e6c0","Type":"ContainerDied","Data":"6ccf22ecc73e2294c4db54ade9dc1f607569d63ea68e96fd0bb072457e4eed8a"}
May 29 11:09:05.349492 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.349460    2547 scope.go:117] "RemoveContainer" containerID="103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4"
May 29 11:09:05.351684 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.351660    2547 generic.go:358] "Generic (PLEG): container finished" podID="88598156-6c72-42b2-842e-88096c72d9dd" containerID="d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56" exitCode=137
May 29 11:09:05.351812 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.351728    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerDied","Data":"d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56"}
May 29 11:09:05.351812 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.351764    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv" event={"ID":"88598156-6c72-42b2-842e-88096c72d9dd","Type":"ContainerDied","Data":"20fbf6753e99f9ff4870236c080f174caf3bcdec1a5ea56a271780ad506ca92c"}
May 29 11:09:05.351812 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.351743    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"
May 29 11:09:05.359139 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.359083    2547 scope.go:117] "RemoveContainer" containerID="ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab"
May 29 11:09:05.368030 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.368011    2547 scope.go:117] "RemoveContainer" containerID="bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253"
May 29 11:09:05.372702 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.372674    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"]
May 29 11:09:05.376218 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.376200    2547 scope.go:117] "RemoveContainer" containerID="42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465"
May 29 11:09:05.377155 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.377096    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6cd86c886d-64d2b"]
May 29 11:09:05.383033 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.383010    2547 scope.go:117] "RemoveContainer" containerID="103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4"
May 29 11:09:05.383268 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.383234    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4\": container with ID starting with 103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4 not found: ID does not exist" containerID="103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4"
May 29 11:09:05.383319 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.383279    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4"} err="failed to get container status \"103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4\": rpc error: code = NotFound desc = could not find container \"103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4\": container with ID starting with 103c3ee1a6207c78a74c74e8a3931ae0fca079ae73b79dd4ba204ce645855be4 not found: ID does not exist"
May 29 11:09:05.383319 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.383297    2547 scope.go:117] "RemoveContainer" containerID="ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab"
May 29 11:09:05.383492 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.383477    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab\": container with ID starting with ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab not found: ID does not exist" containerID="ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab"
May 29 11:09:05.383528 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.383499    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab"} err="failed to get container status \"ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab\": rpc error: code = NotFound desc = could not find container \"ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab\": container with ID starting with ea89eb7672429d939b7477a12205fc7e3e4217e001cb5a3ce14894a628281aab not found: ID does not exist"
May 29 11:09:05.383528 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.383521    2547 scope.go:117] "RemoveContainer" containerID="bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253"
May 29 11:09:05.383720 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.383706    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253\": container with ID starting with bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253 not found: ID does not exist" containerID="bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253"
May 29 11:09:05.383765 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.383723    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253"} err="failed to get container status \"bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253\": rpc error: code = NotFound desc = could not find container \"bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253\": container with ID starting with bb14fa461b323c282f0023456aa5309e591108760f37cf2730763aba8b655253 not found: ID does not exist"
May 29 11:09:05.383765 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.383736    2547 scope.go:117] "RemoveContainer" containerID="42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465"
May 29 11:09:05.383923 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.383907    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465\": container with ID starting with 42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465 not found: ID does not exist" containerID="42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465"
May 29 11:09:05.384032 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.383927    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465"} err="failed to get container status \"42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465\": rpc error: code = NotFound desc = could not find container \"42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465\": container with ID starting with 42c0076dfa165f328c7b80808cd9055522746dd870f0d13f19356031b0350465 not found: ID does not exist"
May 29 11:09:05.384032 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.383940    2547 scope.go:117] "RemoveContainer" containerID="927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7"
May 29 11:09:05.389675 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.389645    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"]
May 29 11:09:05.390871 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.390852    2547 scope.go:117] "RemoveContainer" containerID="d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56"
May 29 11:09:05.393527 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.393506    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-69b65dcc6c-nvddv"]
May 29 11:09:05.399017 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.399000    2547 scope.go:117] "RemoveContainer" containerID="ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05"
May 29 11:09:05.405598 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.405581    2547 scope.go:117] "RemoveContainer" containerID="db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21"
May 29 11:09:05.412132 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.412116    2547 scope.go:117] "RemoveContainer" containerID="927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7"
May 29 11:09:05.412406 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.412386    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7\": container with ID starting with 927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7 not found: ID does not exist" containerID="927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7"
May 29 11:09:05.412470 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.412415    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7"} err="failed to get container status \"927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7\": rpc error: code = NotFound desc = could not find container \"927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7\": container with ID starting with 927bea9e8836d17708a3fcc5f0ecb0133ede8c0e8f46871b4b0bf70d3f157ae7 not found: ID does not exist"
May 29 11:09:05.412470 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.412431    2547 scope.go:117] "RemoveContainer" containerID="d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56"
May 29 11:09:05.412685 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.412666    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56\": container with ID starting with d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56 not found: ID does not exist" containerID="d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56"
May 29 11:09:05.412736 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.412691    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56"} err="failed to get container status \"d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56\": rpc error: code = NotFound desc = could not find container \"d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56\": container with ID starting with d35294728ccea543b5c982ddebd7f5c3b5623f0c1df2fbf0bda861bd22a80e56 not found: ID does not exist"
May 29 11:09:05.412736 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.412707    2547 scope.go:117] "RemoveContainer" containerID="ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05"
May 29 11:09:05.412919 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.412904    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05\": container with ID starting with ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05 not found: ID does not exist" containerID="ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05"
May 29 11:09:05.412958 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.412922    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05"} err="failed to get container status \"ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05\": rpc error: code = NotFound desc = could not find container \"ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05\": container with ID starting with ae9fede6806385f10a16fe41ac82d5a367505962baad7fafe48280255ea90c05 not found: ID does not exist"
May 29 11:09:05.412958 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.412934    2547 scope.go:117] "RemoveContainer" containerID="db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21"
May 29 11:09:05.413145 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.413126    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21\": container with ID starting with db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21 not found: ID does not exist" containerID="db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21"
May 29 11:09:05.413187 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.413149    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21"} err="failed to get container status \"db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21\": rpc error: code = NotFound desc = could not find container \"db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21\": container with ID starting with db8cf86d1213af98a11a523d6891c4bdc423bb36741ba0f6d7a6dd4fe4e4df21 not found: ID does not exist"
May 29 11:09:05.426581 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.426560    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-958sg\" (UniqueName: \"kubernetes.io/projected/fbcad2ea-42a2-4e81-ad33-49abaee247e0-kube-api-access-958sg\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.426664 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.426596    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.426703 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.426661    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.426703 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.426686    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 29 11:09:05.426703 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.426697    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.426821 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.426735    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls podName:fbcad2ea-42a2-4e81-ad33-49abaee247e0 nodeName:}" failed. No retries permitted until 2026-05-29 11:09:05.926717694 +0000 UTC m=+1124.803279900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls") pod "feast-simple-feast-setup-664fcfcc58-5lrg7" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0") : secret "feast-simple-feast-setup-online-tls" not found
May 29 11:09:05.426821 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.426753    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.426821 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.426779    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcad2ea-42a2-4e81-ad33-49abaee247e0-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.426821 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.426814    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 29 11:09:05.427014 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.426855    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-registry-tls podName:fbcad2ea-42a2-4e81-ad33-49abaee247e0 nodeName:}" failed. No retries permitted until 2026-05-29 11:09:05.926843097 +0000 UTC m=+1124.803405318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-5lrg7" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0") : secret "feast-simple-feast-setup-registry-tls" not found
May 29 11:09:05.427014 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.426882    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 29 11:09:05.427014 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.426946    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls podName:fbcad2ea-42a2-4e81-ad33-49abaee247e0 nodeName:}" failed. No retries permitted until 2026-05-29 11:09:05.926926287 +0000 UTC m=+1124.803488509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-5lrg7" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0") : secret "feast-simple-feast-setup-offline-tls" not found
May 29 11:09:05.427119 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.427063    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcad2ea-42a2-4e81-ad33-49abaee247e0-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.428879 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.428860    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.435494 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.435472    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-958sg\" (UniqueName: \"kubernetes.io/projected/fbcad2ea-42a2-4e81-ad33-49abaee247e0-kube-api-access-958sg\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.719330 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.719231    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88598156-6c72-42b2-842e-88096c72d9dd" path="/var/lib/kubelet/pods/88598156-6c72-42b2-842e-88096c72d9dd/volumes"
May 29 11:09:05.719778 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.719759    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff494c0-2142-49d1-aa04-b2574cc5e6c0" path="/var/lib/kubelet/pods/aff494c0-2142-49d1-aa04-b2574cc5e6c0/volumes"
May 29 11:09:05.931204 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.931169    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.931391 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.931217    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.931391 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.931275    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:05.931391 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.931366    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 29 11:09:05.931391 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.931369    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 29 11:09:05.931523 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.931435    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls podName:fbcad2ea-42a2-4e81-ad33-49abaee247e0 nodeName:}" failed. No retries permitted until 2026-05-29 11:09:06.931408 +0000 UTC m=+1125.807970209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls") pod "feast-simple-feast-setup-664fcfcc58-5lrg7" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0") : secret "feast-simple-feast-setup-online-tls" not found
May 29 11:09:05.931523 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:09:05.931450    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls podName:fbcad2ea-42a2-4e81-ad33-49abaee247e0 nodeName:}" failed. No retries permitted until 2026-05-29 11:09:06.931442948 +0000 UTC m=+1125.808005154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-5lrg7" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0") : secret "feast-simple-feast-setup-offline-tls" not found
May 29 11:09:05.933534 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:05.933514    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:06.939272 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:06.939215    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:06.939671 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:06.939315    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:06.941670 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:06.941641    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:06.941795 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:06.941699    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5lrg7\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:07.130218 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:07.130175    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:07.264328 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:07.264301    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"]
May 29 11:09:07.266306 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:09:07.266277    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbcad2ea_42a2_4e81_ad33_49abaee247e0.slice/crio-1484f237f01f4ecf9ecf78e1e0abe66eba078c2905c52224137ee9e999f2a6c1 WatchSource:0}: Error finding container 1484f237f01f4ecf9ecf78e1e0abe66eba078c2905c52224137ee9e999f2a6c1: Status 404 returned error can't find the container with id 1484f237f01f4ecf9ecf78e1e0abe66eba078c2905c52224137ee9e999f2a6c1
May 29 11:09:07.361630 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:07.361593    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerStarted","Data":"1484f237f01f4ecf9ecf78e1e0abe66eba078c2905c52224137ee9e999f2a6c1"}
May 29 11:09:23.417258 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:23.417209    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerStarted","Data":"16bf4776095c3eff3f7b20a5f2fd96e46b82e01d2bf0db6acc06da1cf41f77c1"}
May 29 11:09:27.431150 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:27.431055    2547 generic.go:358] "Generic (PLEG): container finished" podID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerID="16bf4776095c3eff3f7b20a5f2fd96e46b82e01d2bf0db6acc06da1cf41f77c1" exitCode=0
May 29 11:09:27.431150 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:27.431129    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerDied","Data":"16bf4776095c3eff3f7b20a5f2fd96e46b82e01d2bf0db6acc06da1cf41f77c1"}
May 29 11:09:28.435481 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:28.435447    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerStarted","Data":"67831a1ada3abf5b01b01499debb51dc7bf94871f7e4a1048eb81ba6b651c8dd"}
May 29 11:09:33.453253 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:33.453213    2547 generic.go:358] "Generic (PLEG): container finished" podID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerID="67831a1ada3abf5b01b01499debb51dc7bf94871f7e4a1048eb81ba6b651c8dd" exitCode=0
May 29 11:09:33.453644 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:33.453291    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerDied","Data":"67831a1ada3abf5b01b01499debb51dc7bf94871f7e4a1048eb81ba6b651c8dd"}
May 29 11:09:34.460579 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:34.460535    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerStarted","Data":"55f7832b3763811590b4f2396309c46d08711446c05c5b214a0bbba6ac3eb0f2"}
May 29 11:09:34.460579 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:34.460584    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerStarted","Data":"69d23c1a5f6a28b0837d219ddd6237cfae5df5925eb3a699a3804c6f38c497d9"}
May 29 11:09:34.461163 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:34.460599    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerStarted","Data":"eb0ba8e0ed2e9cbe04aac6a3f4626ba6df2454b85bc03bad37f276b2861b099b"}
May 29 11:09:34.461163 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:34.460612    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerStarted","Data":"2d5251f3e1a8df7c06feb7a2e7bfdef4d4b9432e5348be56d154dd74573210c0"}
May 29 11:09:34.484817 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:34.484752    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podStartSLOduration=14.295753007 podStartE2EDuration="29.484733133s" podCreationTimestamp="2026-05-29 11:09:05 +0000 UTC" firstStartedPulling="2026-05-29 11:09:07.268399588 +0000 UTC m=+1126.144961797" lastFinishedPulling="2026-05-29 11:09:22.457379714 +0000 UTC m=+1141.333941923" observedRunningTime="2026-05-29 11:09:34.484268663 +0000 UTC m=+1153.360830889" watchObservedRunningTime="2026-05-29 11:09:34.484733133 +0000 UTC m=+1153.361295362"
May 29 11:09:37.036096 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.036046    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b7cb4cdd7-2r9qs"]
May 29 11:09:37.130543 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.130505    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:37.130743 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.130556    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:37.130743 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.130571    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:37.130743 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.130583    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:37.130743 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.130594    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:37.130743 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.130606    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:37.131433 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.131374    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:37.131433 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.131411    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:37.132726 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.132624    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused"
May 29 11:09:37.132726 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.132640    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 29 11:09:37.132931 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.132715    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused"
May 29 11:09:37.132931 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:37.132803    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 29 11:09:40.131824 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:40.131785    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:40.132350 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:40.132299    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:40.132441 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:40.132429    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:40.132964 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:40.132936    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:40.133071 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:40.133005    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:40.133366 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:40.133344    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:40.136559 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:40.136537    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:09:40.484900 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:09:40.484817    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:10:02.065371 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.065322    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6b7cb4cdd7-2r9qs" podUID="45c11cc2-d291-49d3-86df-1c2e0b258508" containerName="console" containerID="cri-o://9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec" gracePeriod=15
May 29 11:10:02.346515 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.346491    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b7cb4cdd7-2r9qs_45c11cc2-d291-49d3-86df-1c2e0b258508/console/0.log"
May 29 11:10:02.346662 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.346551    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 11:10:02.428613 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.428579    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-oauth-config\") pod \"45c11cc2-d291-49d3-86df-1c2e0b258508\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") "
May 29 11:10:02.428613 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.428620    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-serving-cert\") pod \"45c11cc2-d291-49d3-86df-1c2e0b258508\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") "
May 29 11:10:02.428861 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.428638    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-service-ca\") pod \"45c11cc2-d291-49d3-86df-1c2e0b258508\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") "
May 29 11:10:02.428861 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.428664    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-trusted-ca-bundle\") pod \"45c11cc2-d291-49d3-86df-1c2e0b258508\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") "
May 29 11:10:02.428861 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.428681    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnmxs\" (UniqueName: \"kubernetes.io/projected/45c11cc2-d291-49d3-86df-1c2e0b258508-kube-api-access-hnmxs\") pod \"45c11cc2-d291-49d3-86df-1c2e0b258508\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") "
May 29 11:10:02.428861 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.428694    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-console-config\") pod \"45c11cc2-d291-49d3-86df-1c2e0b258508\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") "
May 29 11:10:02.428861 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.428716    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-oauth-serving-cert\") pod \"45c11cc2-d291-49d3-86df-1c2e0b258508\" (UID: \"45c11cc2-d291-49d3-86df-1c2e0b258508\") "
May 29 11:10:02.429225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.429178    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-service-ca" (OuterVolumeSpecName: "service-ca") pod "45c11cc2-d291-49d3-86df-1c2e0b258508" (UID: "45c11cc2-d291-49d3-86df-1c2e0b258508"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 11:10:02.429225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.429206    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "45c11cc2-d291-49d3-86df-1c2e0b258508" (UID: "45c11cc2-d291-49d3-86df-1c2e0b258508"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 11:10:02.429225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.429228    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-console-config" (OuterVolumeSpecName: "console-config") pod "45c11cc2-d291-49d3-86df-1c2e0b258508" (UID: "45c11cc2-d291-49d3-86df-1c2e0b258508"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 11:10:02.429504 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.429334    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "45c11cc2-d291-49d3-86df-1c2e0b258508" (UID: "45c11cc2-d291-49d3-86df-1c2e0b258508"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 11:10:02.430832 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.430811    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "45c11cc2-d291-49d3-86df-1c2e0b258508" (UID: "45c11cc2-d291-49d3-86df-1c2e0b258508"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:10:02.430922 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.430833    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "45c11cc2-d291-49d3-86df-1c2e0b258508" (UID: "45c11cc2-d291-49d3-86df-1c2e0b258508"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:10:02.430922 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.430844    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c11cc2-d291-49d3-86df-1c2e0b258508-kube-api-access-hnmxs" (OuterVolumeSpecName: "kube-api-access-hnmxs") pod "45c11cc2-d291-49d3-86df-1c2e0b258508" (UID: "45c11cc2-d291-49d3-86df-1c2e0b258508"). InnerVolumeSpecName "kube-api-access-hnmxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:10:02.530021 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.529970    2547 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-oauth-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:10:02.530021 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.530016    2547 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45c11cc2-d291-49d3-86df-1c2e0b258508-console-serving-cert\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:10:02.530021 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.530028    2547 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:10:02.530021 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.530037    2547 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-trusted-ca-bundle\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:10:02.530307 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.530046    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnmxs\" (UniqueName: \"kubernetes.io/projected/45c11cc2-d291-49d3-86df-1c2e0b258508-kube-api-access-hnmxs\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:10:02.530307 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.530056    2547 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-console-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:10:02.530307 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.530066    2547 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45c11cc2-d291-49d3-86df-1c2e0b258508-oauth-serving-cert\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:10:02.551911 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.551887    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b7cb4cdd7-2r9qs_45c11cc2-d291-49d3-86df-1c2e0b258508/console/0.log"
May 29 11:10:02.552030 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.551926    2547 generic.go:358] "Generic (PLEG): container finished" podID="45c11cc2-d291-49d3-86df-1c2e0b258508" containerID="9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec" exitCode=2
May 29 11:10:02.552030 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.551986    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7cb4cdd7-2r9qs"
May 29 11:10:02.552030 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.552008    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7cb4cdd7-2r9qs" event={"ID":"45c11cc2-d291-49d3-86df-1c2e0b258508","Type":"ContainerDied","Data":"9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec"}
May 29 11:10:02.552138 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.552039    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7cb4cdd7-2r9qs" event={"ID":"45c11cc2-d291-49d3-86df-1c2e0b258508","Type":"ContainerDied","Data":"9bbd14394627c7257d37d57d50144965afc95847d36977af902cb715816d45b4"}
May 29 11:10:02.552138 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.552054    2547 scope.go:117] "RemoveContainer" containerID="9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec"
May 29 11:10:02.560331 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.560310    2547 scope.go:117] "RemoveContainer" containerID="9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec"
May 29 11:10:02.560602 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:10:02.560581    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec\": container with ID starting with 9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec not found: ID does not exist" containerID="9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec"
May 29 11:10:02.560653 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.560613    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec"} err="failed to get container status \"9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec\": rpc error: code = NotFound desc = could not find container \"9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec\": container with ID starting with 9358fdae587aafdb031a4057165958a8459ab419fa5d20d71821c9ff15fcfbec not found: ID does not exist"
May 29 11:10:02.572728 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.572700    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b7cb4cdd7-2r9qs"]
May 29 11:10:02.576015 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:02.575992    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b7cb4cdd7-2r9qs"]
May 29 11:10:03.719048 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:03.719014    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c11cc2-d291-49d3-86df-1c2e0b258508" path="/var/lib/kubelet/pods/45c11cc2-d291-49d3-86df-1c2e0b258508/volumes"
May 29 11:10:21.725703 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:21.725666    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 11:10:21.726221 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:21.726197    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 11:10:41.691330 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:41.691281    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"]
May 29 11:10:41.691894 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:41.691715    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="ui" containerID="cri-o://55f7832b3763811590b4f2396309c46d08711446c05c5b214a0bbba6ac3eb0f2" gracePeriod=30
May 29 11:10:41.691894 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:41.691750    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="offline" containerID="cri-o://69d23c1a5f6a28b0837d219ddd6237cfae5df5925eb3a699a3804c6f38c497d9" gracePeriod=30
May 29 11:10:41.691894 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:41.691861    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="online" containerID="cri-o://eb0ba8e0ed2e9cbe04aac6a3f4626ba6df2454b85bc03bad37f276b2861b099b" gracePeriod=30
May 29 11:10:41.692080 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:41.691969    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="registry" containerID="cri-o://2d5251f3e1a8df7c06feb7a2e7bfdef4d4b9432e5348be56d154dd74573210c0" gracePeriod=30
May 29 11:10:42.097782 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.097749    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"]
May 29 11:10:42.098101 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.098087    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c11cc2-d291-49d3-86df-1c2e0b258508" containerName="console"
May 29 11:10:42.098179 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.098103    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c11cc2-d291-49d3-86df-1c2e0b258508" containerName="console"
May 29 11:10:42.098225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.098184    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="45c11cc2-d291-49d3-86df-1c2e0b258508" containerName="console"
May 29 11:10:42.100708 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.100689    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.103161 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.103138    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bkmmt\""
May 29 11:10:42.114109 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.114026    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"]
May 29 11:10:42.251208 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.251167    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.251415 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.251233    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.251415 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.251292    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.251415 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.251309    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wlst\" (UniqueName: \"kubernetes.io/projected/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-kube-api-access-4wlst\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.251415 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.251340    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.251415 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.251368    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.352419 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.352354    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.352419 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.352397    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.352419 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.352415    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4wlst\" (UniqueName: \"kubernetes.io/projected/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-kube-api-access-4wlst\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.352643 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.352543    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.352643 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.352596    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.352748 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.352688    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.352799 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:10:42.352750    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 29 11:10:42.352853 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:10:42.352815    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-online-tls podName:f5a72401-9ca3-4dd8-b9b4-8293a4de0cea nodeName:}" failed. No retries permitted until 2026-05-29 11:10:42.852795162 +0000 UTC m=+1221.729357381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-online-tls") pod "feast-simple-feast-setup-664fcfcc58-srz5t" (UID: "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea") : secret "feast-simple-feast-setup-online-tls" not found
May 29 11:10:42.353191 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.353117    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.355065 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.355035    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.355166 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.355129    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.355209 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.355198    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.360878 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.360855    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wlst\" (UniqueName: \"kubernetes.io/projected/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-kube-api-access-4wlst\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.677910 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.677828    2547 generic.go:358] "Generic (PLEG): container finished" podID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerID="55f7832b3763811590b4f2396309c46d08711446c05c5b214a0bbba6ac3eb0f2" exitCode=0
May 29 11:10:42.677910 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.677851    2547 generic.go:358] "Generic (PLEG): container finished" podID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerID="eb0ba8e0ed2e9cbe04aac6a3f4626ba6df2454b85bc03bad37f276b2861b099b" exitCode=0
May 29 11:10:42.678096 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.677908    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerDied","Data":"55f7832b3763811590b4f2396309c46d08711446c05c5b214a0bbba6ac3eb0f2"}
May 29 11:10:42.678096 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.677946    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerDied","Data":"eb0ba8e0ed2e9cbe04aac6a3f4626ba6df2454b85bc03bad37f276b2861b099b"}
May 29 11:10:42.858127 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.858096    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:42.860346 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:42.860322    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-srz5t\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:43.012770 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:43.012737    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:43.146709 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:43.146680    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"]
May 29 11:10:43.149312 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:10:43.149283    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5a72401_9ca3_4dd8_b9b4_8293a4de0cea.slice/crio-08524b6a8185ea37eb9d291993677db3e589262f5b33bf4c41954fe004eb23cf WatchSource:0}: Error finding container 08524b6a8185ea37eb9d291993677db3e589262f5b33bf4c41954fe004eb23cf: Status 404 returned error can't find the container with id 08524b6a8185ea37eb9d291993677db3e589262f5b33bf4c41954fe004eb23cf
May 29 11:10:43.681877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:43.681835    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerStarted","Data":"c83b91bd58be46df8b8b3a268e97b88465db9c97a1458d91dea37eca66f00e59"}
May 29 11:10:43.681877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:43.681876    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerStarted","Data":"08524b6a8185ea37eb9d291993677db3e589262f5b33bf4c41954fe004eb23cf"}
May 29 11:10:48.697660 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:48.697627    2547 generic.go:358] "Generic (PLEG): container finished" podID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerID="c83b91bd58be46df8b8b3a268e97b88465db9c97a1458d91dea37eca66f00e59" exitCode=0
May 29 11:10:48.698041 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:48.697681    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerDied","Data":"c83b91bd58be46df8b8b3a268e97b88465db9c97a1458d91dea37eca66f00e59"}
May 29 11:10:49.703199 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:49.703163    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerStarted","Data":"cb54481ba7c029816ccc23cbc495e7ebabafd6d6c9e2be5862b057a9ddd8a5ad"}
May 29 11:10:50.133348 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:50.133299    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 29 11:10:50.482449 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:50.482360    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 29 11:10:54.719785 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:54.719751    2547 generic.go:358] "Generic (PLEG): container finished" podID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerID="cb54481ba7c029816ccc23cbc495e7ebabafd6d6c9e2be5862b057a9ddd8a5ad" exitCode=0
May 29 11:10:54.720152 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:54.719810    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerDied","Data":"cb54481ba7c029816ccc23cbc495e7ebabafd6d6c9e2be5862b057a9ddd8a5ad"}
May 29 11:10:55.727627 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:55.727578    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerStarted","Data":"dd54c36893814d3f504284161d7a56933e95ef9316e81f1cf7a15aaa316e7ca3"}
May 29 11:10:55.728087 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:55.727637    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerStarted","Data":"8208c3052b0b90c98dc4a4f0b78637d01cd53b9a7b62cffc7702ae77ae9395bb"}
May 29 11:10:55.728087 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:55.727653    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerStarted","Data":"e5a3821ac6fee2f7682d15a0d5cc2ae7a9f0e11c86ef643eac90cadf8ade17ef"}
May 29 11:10:55.728087 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:55.727666    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerStarted","Data":"301ea4564bc10cc35bf2651c52df32e52911ae88beb701fe1bbbfc1d00454d84"}
May 29 11:10:55.753670 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:55.753600    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podStartSLOduration=13.753580972 podStartE2EDuration="13.753580972s" podCreationTimestamp="2026-05-29 11:10:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:10:55.750458753 +0000 UTC m=+1234.627020982" watchObservedRunningTime="2026-05-29 11:10:55.753580972 +0000 UTC m=+1234.630143201"
May 29 11:10:58.013132 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:58.013075    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:58.013979 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:58.013895    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:58.013979 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:58.013944    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:58.013979 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:58.013957    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:10:58.014614 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:58.014583    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused"
May 29 11:10:58.014955 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:58.014914    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 29 11:10:58.015080 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:58.014951    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 29 11:10:58.015080 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:10:58.014963    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused"
May 29 11:11:00.132980 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:00.132913    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 29 11:11:00.483033 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:00.482929    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 29 11:11:01.014061 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.014033    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.014269 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.014121    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.014442 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.014421    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.014901 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.014882    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.015034 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.014906    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.015034 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.014921    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.015256 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.015213    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.015368 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.015346    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.015433 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.015384    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.018679 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.018662    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.749839 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.749792    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:01.752713 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:01.752692    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:11:10.133308 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:10.133185    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 29 11:11:10.133733 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:10.133364    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:11:10.482102 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:10.481995    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 29 11:11:10.482289 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:10.482144    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:11:11.781252 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:11.781210    2547 generic.go:358] "Generic (PLEG): container finished" podID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerID="69d23c1a5f6a28b0837d219ddd6237cfae5df5925eb3a699a3804c6f38c497d9" exitCode=137
May 29 11:11:11.781252 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:11.781234    2547 generic.go:358] "Generic (PLEG): container finished" podID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerID="2d5251f3e1a8df7c06feb7a2e7bfdef4d4b9432e5348be56d154dd74573210c0" exitCode=137
May 29 11:11:11.781613 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:11.781266    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerDied","Data":"69d23c1a5f6a28b0837d219ddd6237cfae5df5925eb3a699a3804c6f38c497d9"}
May 29 11:11:11.781613 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:11.781299    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerDied","Data":"2d5251f3e1a8df7c06feb7a2e7bfdef4d4b9432e5348be56d154dd74573210c0"}
May 29 11:11:12.338314 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.338291    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:11:12.409325 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.409294    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcad2ea-42a2-4e81-ad33-49abaee247e0-feast-data\") pod \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") "
May 29 11:11:12.409481 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.409338    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls\") pod \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") "
May 29 11:11:12.409481 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.409369    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-ui-tls\") pod \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") "
May 29 11:11:12.409481 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.409388    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-958sg\" (UniqueName: \"kubernetes.io/projected/fbcad2ea-42a2-4e81-ad33-49abaee247e0-kube-api-access-958sg\") pod \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") "
May 29 11:11:12.409481 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.409414    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-registry-tls\") pod \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") "
May 29 11:11:12.409481 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.409447    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls\") pod \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\" (UID: \"fbcad2ea-42a2-4e81-ad33-49abaee247e0\") "
May 29 11:11:12.409917 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.409881    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcad2ea-42a2-4e81-ad33-49abaee247e0-feast-data" (OuterVolumeSpecName: "feast-data") pod "fbcad2ea-42a2-4e81-ad33-49abaee247e0" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:11:12.411808 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.411781    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls" (OuterVolumeSpecName: "online-tls") pod "fbcad2ea-42a2-4e81-ad33-49abaee247e0" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:11:12.412084 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.412067    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "fbcad2ea-42a2-4e81-ad33-49abaee247e0" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:11:12.412145 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.412103    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcad2ea-42a2-4e81-ad33-49abaee247e0-kube-api-access-958sg" (OuterVolumeSpecName: "kube-api-access-958sg") pod "fbcad2ea-42a2-4e81-ad33-49abaee247e0" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0"). InnerVolumeSpecName "kube-api-access-958sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:11:12.412217 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.412202    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "fbcad2ea-42a2-4e81-ad33-49abaee247e0" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:11:12.412334 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.412231    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fbcad2ea-42a2-4e81-ad33-49abaee247e0" (UID: "fbcad2ea-42a2-4e81-ad33-49abaee247e0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:11:12.510948 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.510849    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcad2ea-42a2-4e81-ad33-49abaee247e0-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:11:12.510948 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.510886    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:11:12.510948 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.510898    2547 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-ui-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:11:12.510948 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.510912    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-958sg\" (UniqueName: \"kubernetes.io/projected/fbcad2ea-42a2-4e81-ad33-49abaee247e0-kube-api-access-958sg\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:11:12.510948 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.510928    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:11:12.510948 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.510940    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fbcad2ea-42a2-4e81-ad33-49abaee247e0-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:11:12.787623 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.787596    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"
May 29 11:11:12.788046 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.787595    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7" event={"ID":"fbcad2ea-42a2-4e81-ad33-49abaee247e0","Type":"ContainerDied","Data":"1484f237f01f4ecf9ecf78e1e0abe66eba078c2905c52224137ee9e999f2a6c1"}
May 29 11:11:12.788046 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.787725    2547 scope.go:117] "RemoveContainer" containerID="55f7832b3763811590b4f2396309c46d08711446c05c5b214a0bbba6ac3eb0f2"
May 29 11:11:12.796303 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.796284    2547 scope.go:117] "RemoveContainer" containerID="69d23c1a5f6a28b0837d219ddd6237cfae5df5925eb3a699a3804c6f38c497d9"
May 29 11:11:12.803981 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.803962    2547 scope.go:117] "RemoveContainer" containerID="eb0ba8e0ed2e9cbe04aac6a3f4626ba6df2454b85bc03bad37f276b2861b099b"
May 29 11:11:12.811404 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.811370    2547 scope.go:117] "RemoveContainer" containerID="2d5251f3e1a8df7c06feb7a2e7bfdef4d4b9432e5348be56d154dd74573210c0"
May 29 11:11:12.811547 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.811522    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"]
May 29 11:11:12.815129 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.815105    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5lrg7"]
May 29 11:11:12.819770 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.819756    2547 scope.go:117] "RemoveContainer" containerID="67831a1ada3abf5b01b01499debb51dc7bf94871f7e4a1048eb81ba6b651c8dd"
May 29 11:11:12.826711 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:12.826694    2547 scope.go:117] "RemoveContainer" containerID="16bf4776095c3eff3f7b20a5f2fd96e46b82e01d2bf0db6acc06da1cf41f77c1"
May 29 11:11:13.721401 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:11:13.721368    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" path="/var/lib/kubelet/pods/fbcad2ea-42a2-4e81-ad33-49abaee247e0/volumes"
May 29 11:12:03.706506 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.706472    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"]
May 29 11:12:03.707964 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.707934    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="feast-init"
May 29 11:12:03.707964 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.707964    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="feast-init"
May 29 11:12:03.707964 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.707977    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="registry"
May 29 11:12:03.708276 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.707988    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="registry"
May 29 11:12:03.708276 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708010    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="feast-apply"
May 29 11:12:03.708276 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708017    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="feast-apply"
May 29 11:12:03.708276 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708059    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="ui"
May 29 11:12:03.708276 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708068    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="ui"
May 29 11:12:03.708276 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708084    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="offline"
May 29 11:12:03.708276 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708092    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="offline"
May 29 11:12:03.708276 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708104    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="online"
May 29 11:12:03.708276 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708114    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="online"
May 29 11:12:03.708659 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708294    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="offline"
May 29 11:12:03.708659 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708320    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="registry"
May 29 11:12:03.708659 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708333    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="online"
May 29 11:12:03.708659 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.708351    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbcad2ea-42a2-4e81-ad33-49abaee247e0" containerName="ui"
May 29 11:12:03.713562 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.713537    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.716424 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.716400    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 29 11:12:03.716619 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.716411    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 29 11:12:03.716730 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.716431    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 29 11:12:03.716730 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.716431    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-wjbzq\""
May 29 11:12:03.716730 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.716431    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 29 11:12:03.716730 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.716466    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 29 11:12:03.717133 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.716999    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 29 11:12:03.723209 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.723187    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"]
May 29 11:12:03.838044 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.838008    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85zn\" (UniqueName: \"kubernetes.io/projected/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-kube-api-access-m85zn\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.838228 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.838055    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.838228 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.838076    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.838228 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.838157    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.838228 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.838189    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.838228 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.838207    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.938797 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.938751    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.938797 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.938799    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.939058 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.938827    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.939058 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.938914    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m85zn\" (UniqueName: \"kubernetes.io/projected/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-kube-api-access-m85zn\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.939058 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.938965    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.939058 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.938990    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.939471 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.939443    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.939636 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.939612    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.941549 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.941528    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.941762 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.941743    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.941816 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.941772    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:03.947840 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:03.947817    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85zn\" (UniqueName: \"kubernetes.io/projected/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-kube-api-access-m85zn\") pod \"feast-simple-feast-remote-setup-5cff94746b-ggvjp\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:04.023751 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:04.023723    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:04.164016 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:04.163990    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"]
May 29 11:12:04.166227 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:12:04.166200    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d7ee012_b1c5_45a5_8868_9f119fa1f68d.slice/crio-92b33e93e0e7e5f3d5b90c5baff9518d7b83f290b9b14ef3041221c06cd400dd WatchSource:0}: Error finding container 92b33e93e0e7e5f3d5b90c5baff9518d7b83f290b9b14ef3041221c06cd400dd: Status 404 returned error can't find the container with id 92b33e93e0e7e5f3d5b90c5baff9518d7b83f290b9b14ef3041221c06cd400dd
May 29 11:12:04.946406 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:04.946365    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerStarted","Data":"14144a4c9e2fe9bfb6d91bac91279a43882e15b28410b27f0bd4fe3c34250811"}
May 29 11:12:04.946806 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:04.946410    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerStarted","Data":"92b33e93e0e7e5f3d5b90c5baff9518d7b83f290b9b14ef3041221c06cd400dd"}
May 29 11:12:08.958592 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:08.958557    2547 generic.go:358] "Generic (PLEG): container finished" podID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerID="14144a4c9e2fe9bfb6d91bac91279a43882e15b28410b27f0bd4fe3c34250811" exitCode=0
May 29 11:12:08.958998 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:08.958630    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerDied","Data":"14144a4c9e2fe9bfb6d91bac91279a43882e15b28410b27f0bd4fe3c34250811"}
May 29 11:12:09.963824 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:09.963789    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerStarted","Data":"c246bf4eb8bf3f8ab16805a3dea5c341d3661dc26fa11e0a271a3068cca247b0"}
May 29 11:12:14.981195 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:14.981113    2547 generic.go:358] "Generic (PLEG): container finished" podID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerID="c246bf4eb8bf3f8ab16805a3dea5c341d3661dc26fa11e0a271a3068cca247b0" exitCode=0
May 29 11:12:14.981195 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:14.981183    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerDied","Data":"c246bf4eb8bf3f8ab16805a3dea5c341d3661dc26fa11e0a271a3068cca247b0"}
May 29 11:12:15.988784 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:15.988746    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerStarted","Data":"9e6e777b454ee3bf40ce80d2cf7eac3a865ae52e5159452dd48eac948c829fcc"}
May 29 11:12:15.988784 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:15.988782    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerStarted","Data":"7699b1a19f73afebcdb644f6e5f5875d26c22156f5dc8cee006061e359b7f22c"}
May 29 11:12:15.988784 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:15.988791    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerStarted","Data":"c7ece10c2c780b7c2c83813391c3d8d7621cb0025069c801bc3cbfca0a6cdf87"}
May 29 11:12:16.014720 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:16.013064    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podStartSLOduration=13.01304268 podStartE2EDuration="13.01304268s" podCreationTimestamp="2026-05-29 11:12:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:12:16.0111918 +0000 UTC m=+1314.887754028" watchObservedRunningTime="2026-05-29 11:12:16.01304268 +0000 UTC m=+1314.889604909"
May 29 11:12:16.024405 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:16.024279    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:16.024405 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:16.024334    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:16.024405 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:16.024350    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:16.026167 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:16.026122    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused"
May 29 11:12:16.026167 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:16.026147    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused"
May 29 11:12:16.026414 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:16.026188    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused"
May 29 11:12:19.024804 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:19.024749    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused"
May 29 11:12:19.024804 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:19.024780    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused"
May 29 11:12:19.025393 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:19.024834    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused"
May 29 11:12:22.024965 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:22.024931    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:22.025395 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:22.025294    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:22.025558 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:22.025535    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:22.025849 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:22.025828    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:22.029009 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:22.028989    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:23.011551 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:23.011515    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:23.011551 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:23.011551    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:23.012006 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:23.011990    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:12:23.014623 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:12:23.014598    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:13:25.065570 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:25.065517    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"]
May 29 11:13:25.066182 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:25.065929    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="online" containerID="cri-o://c7ece10c2c780b7c2c83813391c3d8d7621cb0025069c801bc3cbfca0a6cdf87" gracePeriod=30
May 29 11:13:25.066182 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:25.065989    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="ui" containerID="cri-o://9e6e777b454ee3bf40ce80d2cf7eac3a865ae52e5159452dd48eac948c829fcc" gracePeriod=30
May 29 11:13:25.066182 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:25.065988    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="offline" containerID="cri-o://7699b1a19f73afebcdb644f6e5f5875d26c22156f5dc8cee006061e359b7f22c" gracePeriod=30
May 29 11:13:25.209797 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:25.209760    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"]
May 29 11:13:25.210296 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:25.210222    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="registry" containerID="cri-o://301ea4564bc10cc35bf2651c52df32e52911ae88beb701fe1bbbfc1d00454d84" gracePeriod=30
May 29 11:13:25.210452 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:25.210307    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="offline" containerID="cri-o://8208c3052b0b90c98dc4a4f0b78637d01cd53b9a7b62cffc7702ae77ae9395bb" gracePeriod=30
May 29 11:13:25.210452 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:25.210309    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="online" containerID="cri-o://e5a3821ac6fee2f7682d15a0d5cc2ae7a9f0e11c86ef643eac90cadf8ade17ef" gracePeriod=30
May 29 11:13:25.210452 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:25.210328    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="ui" containerID="cri-o://dd54c36893814d3f504284161d7a56933e95ef9316e81f1cf7a15aaa316e7ca3" gracePeriod=30
May 29 11:13:26.199035 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:26.198949    2547 generic.go:358] "Generic (PLEG): container finished" podID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerID="dd54c36893814d3f504284161d7a56933e95ef9316e81f1cf7a15aaa316e7ca3" exitCode=0
May 29 11:13:26.199035 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:26.198976    2547 generic.go:358] "Generic (PLEG): container finished" podID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerID="e5a3821ac6fee2f7682d15a0d5cc2ae7a9f0e11c86ef643eac90cadf8ade17ef" exitCode=0
May 29 11:13:26.199035 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:26.199020    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerDied","Data":"dd54c36893814d3f504284161d7a56933e95ef9316e81f1cf7a15aaa316e7ca3"}
May 29 11:13:26.199564 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:26.199055    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerDied","Data":"e5a3821ac6fee2f7682d15a0d5cc2ae7a9f0e11c86ef643eac90cadf8ade17ef"}
May 29 11:13:26.201352 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:26.201328    2547 generic.go:358] "Generic (PLEG): container finished" podID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerID="9e6e777b454ee3bf40ce80d2cf7eac3a865ae52e5159452dd48eac948c829fcc" exitCode=0
May 29 11:13:26.201352 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:26.201349    2547 generic.go:358] "Generic (PLEG): container finished" podID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerID="c7ece10c2c780b7c2c83813391c3d8d7621cb0025069c801bc3cbfca0a6cdf87" exitCode=0
May 29 11:13:26.201352 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:26.201339    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerDied","Data":"9e6e777b454ee3bf40ce80d2cf7eac3a865ae52e5159452dd48eac948c829fcc"}
May 29 11:13:26.201543 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:26.201378    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerDied","Data":"c7ece10c2c780b7c2c83813391c3d8d7621cb0025069c801bc3cbfca0a6cdf87"}
May 29 11:13:31.015202 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:31.015147    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 29 11:13:31.750953 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:31.750906    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 29 11:13:32.025778 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:32.025743    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused"
May 29 11:13:33.012216 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:33.012170    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused"
May 29 11:13:41.015004 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:41.014957    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 29 11:13:41.750201 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:41.750162    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 29 11:13:42.026263 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:42.026206    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused"
May 29 11:13:43.011916 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:43.011867    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused"
May 29 11:13:51.015054 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:51.015004    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 29 11:13:51.015550 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:51.015179    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:13:51.750617 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:51.750571    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 29 11:13:51.750785 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:51.750694    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:13:52.025854 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:52.025822    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused"
May 29 11:13:52.026221 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:52.025928    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:13:53.012344 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:53.012295    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused"
May 29 11:13:53.012532 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:53.012456    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:13:55.288654 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.288624    2547 generic.go:358] "Generic (PLEG): container finished" podID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerID="8208c3052b0b90c98dc4a4f0b78637d01cd53b9a7b62cffc7702ae77ae9395bb" exitCode=137
May 29 11:13:55.288654 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.288649    2547 generic.go:358] "Generic (PLEG): container finished" podID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerID="301ea4564bc10cc35bf2651c52df32e52911ae88beb701fe1bbbfc1d00454d84" exitCode=137
May 29 11:13:55.289046 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.288699    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerDied","Data":"8208c3052b0b90c98dc4a4f0b78637d01cd53b9a7b62cffc7702ae77ae9395bb"}
May 29 11:13:55.289046 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.288739    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerDied","Data":"301ea4564bc10cc35bf2651c52df32e52911ae88beb701fe1bbbfc1d00454d84"}
May 29 11:13:55.290774 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.290754    2547 generic.go:358] "Generic (PLEG): container finished" podID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerID="7699b1a19f73afebcdb644f6e5f5875d26c22156f5dc8cee006061e359b7f22c" exitCode=137
May 29 11:13:55.290881 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.290822    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerDied","Data":"7699b1a19f73afebcdb644f6e5f5875d26c22156f5dc8cee006061e359b7f22c"}
May 29 11:13:55.290881 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.290847    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp" event={"ID":"3d7ee012-b1c5-45a5-8868-9f119fa1f68d","Type":"ContainerDied","Data":"92b33e93e0e7e5f3d5b90c5baff9518d7b83f290b9b14ef3041221c06cd400dd"}
May 29 11:13:55.290881 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.290857    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b33e93e0e7e5f3d5b90c5baff9518d7b83f290b9b14ef3041221c06cd400dd"
May 29 11:13:55.319831 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.319813    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:13:55.399310 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.399200    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-offline-tls\") pod \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") "
May 29 11:13:55.399310 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.399234    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-online-tls\") pod \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") "
May 29 11:13:55.399543 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.399403    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-registry-tls\") pod \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") "
May 29 11:13:55.399543 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.399443    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-ui-tls\") pod \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") "
May 29 11:13:55.399543 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.399476    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m85zn\" (UniqueName: \"kubernetes.io/projected/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-kube-api-access-m85zn\") pod \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") "
May 29 11:13:55.399543 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.399537    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-feast-data\") pod \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\" (UID: \"3d7ee012-b1c5-45a5-8868-9f119fa1f68d\") "
May 29 11:13:55.399792 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.399766    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3d7ee012-b1c5-45a5-8868-9f119fa1f68d" (UID: "3d7ee012-b1c5-45a5-8868-9f119fa1f68d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 11:13:55.400124 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.400094    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-feast-data" (OuterVolumeSpecName: "feast-data") pod "3d7ee012-b1c5-45a5-8868-9f119fa1f68d" (UID: "3d7ee012-b1c5-45a5-8868-9f119fa1f68d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:13:55.401597 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.401572    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "3d7ee012-b1c5-45a5-8868-9f119fa1f68d" (UID: "3d7ee012-b1c5-45a5-8868-9f119fa1f68d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:13:55.401691 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.401574    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-kube-api-access-m85zn" (OuterVolumeSpecName: "kube-api-access-m85zn") pod "3d7ee012-b1c5-45a5-8868-9f119fa1f68d" (UID: "3d7ee012-b1c5-45a5-8868-9f119fa1f68d"). InnerVolumeSpecName "kube-api-access-m85zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:13:55.401691 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.401629    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-online-tls" (OuterVolumeSpecName: "online-tls") pod "3d7ee012-b1c5-45a5-8868-9f119fa1f68d" (UID: "3d7ee012-b1c5-45a5-8868-9f119fa1f68d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:13:55.401691 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.401644    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "3d7ee012-b1c5-45a5-8868-9f119fa1f68d" (UID: "3d7ee012-b1c5-45a5-8868-9f119fa1f68d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:13:55.500373 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.500329    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:55.500373 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.500366    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:55.500373 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.500380    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:55.500608 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.500391    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:55.500608 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.500403    2547 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-ui-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:55.500608 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.500414    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m85zn\" (UniqueName: \"kubernetes.io/projected/3d7ee012-b1c5-45a5-8868-9f119fa1f68d-kube-api-access-m85zn\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:55.848613 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.848588    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:13:55.903692 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.903662    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-ui-tls\") pod \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") "
May 29 11:13:55.903870 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.903718    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wlst\" (UniqueName: \"kubernetes.io/projected/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-kube-api-access-4wlst\") pod \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") "
May 29 11:13:55.903870 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.903801    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-feast-data\") pod \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") "
May 29 11:13:55.903870 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.903825    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-offline-tls\") pod \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") "
May 29 11:13:55.903870 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.903854    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-online-tls\") pod \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") "
May 29 11:13:55.904085 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.903882    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-registry-tls\") pod \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\" (UID: \"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea\") "
May 29 11:13:55.904416 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.904383    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-feast-data" (OuterVolumeSpecName: "feast-data") pod "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" (UID: "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:13:55.905925 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.905896    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" (UID: "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:13:55.906029 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.905953    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-kube-api-access-4wlst" (OuterVolumeSpecName: "kube-api-access-4wlst") pod "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" (UID: "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea"). InnerVolumeSpecName "kube-api-access-4wlst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:13:55.906029 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.905951    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" (UID: "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:13:55.906233 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.906217    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" (UID: "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:13:55.906233 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:55.906252    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-online-tls" (OuterVolumeSpecName: "online-tls") pod "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" (UID: "f5a72401-9ca3-4dd8-b9b4-8293a4de0cea"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:13:56.004772 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.004692    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:56.004772 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.004719    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:56.004772 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.004728    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:56.004772 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.004736    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:56.004772 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.004744    2547 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-ui-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:56.004772 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.004754    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wlst\" (UniqueName: \"kubernetes.io/projected/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea-kube-api-access-4wlst\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:13:56.296847 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.296821    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"
May 29 11:13:56.297208 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.296849    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"
May 29 11:13:56.297208 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.296848    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t" event={"ID":"f5a72401-9ca3-4dd8-b9b4-8293a4de0cea","Type":"ContainerDied","Data":"08524b6a8185ea37eb9d291993677db3e589262f5b33bf4c41954fe004eb23cf"}
May 29 11:13:56.297208 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.296893    2547 scope.go:117] "RemoveContainer" containerID="dd54c36893814d3f504284161d7a56933e95ef9316e81f1cf7a15aaa316e7ca3"
May 29 11:13:56.306109 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.306090    2547 scope.go:117] "RemoveContainer" containerID="8208c3052b0b90c98dc4a4f0b78637d01cd53b9a7b62cffc7702ae77ae9395bb"
May 29 11:13:56.313321 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.313286    2547 scope.go:117] "RemoveContainer" containerID="e5a3821ac6fee2f7682d15a0d5cc2ae7a9f0e11c86ef643eac90cadf8ade17ef"
May 29 11:13:56.315625 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.315601    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"]
May 29 11:13:56.318651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.318629    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-ggvjp"]
May 29 11:13:56.321107 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.321088    2547 scope.go:117] "RemoveContainer" containerID="301ea4564bc10cc35bf2651c52df32e52911ae88beb701fe1bbbfc1d00454d84"
May 29 11:13:56.334337 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.328882    2547 scope.go:117] "RemoveContainer" containerID="cb54481ba7c029816ccc23cbc495e7ebabafd6d6c9e2be5862b057a9ddd8a5ad"
May 29 11:13:56.334337 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.334263    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"]
May 29 11:13:56.336489 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.336467    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-srz5t"]
May 29 11:13:56.341832 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:56.341808    2547 scope.go:117] "RemoveContainer" containerID="c83b91bd58be46df8b8b3a268e97b88465db9c97a1458d91dea37eca66f00e59"
May 29 11:13:57.717487 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:57.717454    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" path="/var/lib/kubelet/pods/3d7ee012-b1c5-45a5-8868-9f119fa1f68d/volumes"
May 29 11:13:57.717998 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:13:57.717983    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" path="/var/lib/kubelet/pods/f5a72401-9ca3-4dd8-b9b4-8293a4de0cea/volumes"
May 29 11:14:14.092393 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.092304    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"]
May 29 11:14:14.092921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.092531    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v" podUID="dc4a531e-48d9-4811-8344-e176e5bcdd65" containerName="manager" containerID="cri-o://b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b" gracePeriod=10
May 29 11:14:14.333146 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.333122    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"
May 29 11:14:14.361513 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.361430    2547 generic.go:358] "Generic (PLEG): container finished" podID="dc4a531e-48d9-4811-8344-e176e5bcdd65" containerID="b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b" exitCode=0
May 29 11:14:14.361513 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.361493    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"
May 29 11:14:14.361705 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.361504    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v" event={"ID":"dc4a531e-48d9-4811-8344-e176e5bcdd65","Type":"ContainerDied","Data":"b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b"}
May 29 11:14:14.361705 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.361545    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-4794v" event={"ID":"dc4a531e-48d9-4811-8344-e176e5bcdd65","Type":"ContainerDied","Data":"ce96e4d0d933893161c95b6be0ffa7160c3ed5cc397a494b80ae67c4f2c2ae17"}
May 29 11:14:14.361705 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.361563    2547 scope.go:117] "RemoveContainer" containerID="b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b"
May 29 11:14:14.371545 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.371527    2547 scope.go:117] "RemoveContainer" containerID="b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b"
May 29 11:14:14.371876 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:14:14.371839    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b\": container with ID starting with b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b not found: ID does not exist" containerID="b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b"
May 29 11:14:14.371970 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.371886    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b"} err="failed to get container status \"b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b\": rpc error: code = NotFound desc = could not find container \"b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b\": container with ID starting with b052de8b6fca1a2a5ad0863f25dbc8bab2362013614674c72989dd6855e3089b not found: ID does not exist"
May 29 11:14:14.455365 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.455336    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7blp\" (UniqueName: \"kubernetes.io/projected/dc4a531e-48d9-4811-8344-e176e5bcdd65-kube-api-access-d7blp\") pod \"dc4a531e-48d9-4811-8344-e176e5bcdd65\" (UID: \"dc4a531e-48d9-4811-8344-e176e5bcdd65\") "
May 29 11:14:14.457359 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.457340    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4a531e-48d9-4811-8344-e176e5bcdd65-kube-api-access-d7blp" (OuterVolumeSpecName: "kube-api-access-d7blp") pod "dc4a531e-48d9-4811-8344-e176e5bcdd65" (UID: "dc4a531e-48d9-4811-8344-e176e5bcdd65"). InnerVolumeSpecName "kube-api-access-d7blp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:14:14.555986 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.555959    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7blp\" (UniqueName: \"kubernetes.io/projected/dc4a531e-48d9-4811-8344-e176e5bcdd65-kube-api-access-d7blp\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:14:14.681967 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.681931    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"]
May 29 11:14:14.683755 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:14.683729    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-4794v"]
May 29 11:14:15.717069 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:15.717037    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4a531e-48d9-4811-8344-e176e5bcdd65" path="/var/lib/kubelet/pods/dc4a531e-48d9-4811-8344-e176e5bcdd65/volumes"
May 29 11:14:18.388024 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.387987    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"]
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388322    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="registry"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388335    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="registry"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388346    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="feast-init"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388352    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="feast-init"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388360    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="online"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388366    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="online"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388372    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="ui"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388378    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="ui"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388387    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="offline"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388392    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="offline"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388405    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="feast-apply"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388410    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="feast-apply"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388418    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="ui"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388423    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="ui"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388429    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="feast-apply"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388433    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="feast-apply"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388440    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="offline"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388445    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="offline"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388450    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="online"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388455    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="online"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388461    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="feast-init"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388467    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="feast-init"
May 29 11:14:18.388464 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388477    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4a531e-48d9-4811-8344-e176e5bcdd65" containerName="manager"
May 29 11:14:18.389422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388482    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4a531e-48d9-4811-8344-e176e5bcdd65" containerName="manager"
May 29 11:14:18.389422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388523    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="online"
May 29 11:14:18.389422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388531    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="ui"
May 29 11:14:18.389422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388537    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="offline"
May 29 11:14:18.389422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388544    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="ui"
May 29 11:14:18.389422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388552    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="registry"
May 29 11:14:18.389422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388560    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5a72401-9ca3-4dd8-b9b4-8293a4de0cea" containerName="offline"
May 29 11:14:18.389422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388565    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d7ee012-b1c5-45a5-8868-9f119fa1f68d" containerName="online"
May 29 11:14:18.389422 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.388571    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc4a531e-48d9-4811-8344-e176e5bcdd65" containerName="manager"
May 29 11:14:18.394840 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.393741    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"
May 29 11:14:18.397585 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.397430    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 29 11:14:18.397585 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.397462    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-zh8ts\""
May 29 11:14:18.397585 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.397491    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 29 11:14:18.399193 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.399165    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"]
May 29 11:14:18.488285 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.488219    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwp7t\" (UniqueName: \"kubernetes.io/projected/379b1eee-3ab7-4510-b811-92c0536ca24c-kube-api-access-bwp7t\") pod \"feast-operator-controller-manager-cf99db75-2qld6\" (UID: \"379b1eee-3ab7-4510-b811-92c0536ca24c\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"
May 29 11:14:18.588982 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.588943    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bwp7t\" (UniqueName: \"kubernetes.io/projected/379b1eee-3ab7-4510-b811-92c0536ca24c-kube-api-access-bwp7t\") pod \"feast-operator-controller-manager-cf99db75-2qld6\" (UID: \"379b1eee-3ab7-4510-b811-92c0536ca24c\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"
May 29 11:14:18.597044 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.597018    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwp7t\" (UniqueName: \"kubernetes.io/projected/379b1eee-3ab7-4510-b811-92c0536ca24c-kube-api-access-bwp7t\") pod \"feast-operator-controller-manager-cf99db75-2qld6\" (UID: \"379b1eee-3ab7-4510-b811-92c0536ca24c\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"
May 29 11:14:18.706491 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.706393    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"
May 29 11:14:18.839886 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.839843    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"]
May 29 11:14:18.843118 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:14:18.843090    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379b1eee_3ab7_4510_b811_92c0536ca24c.slice/crio-2183a781098a57d77f11a3cfc26d18cae759cfc6de2f0a321ef8ac9d9cb42ef8 WatchSource:0}: Error finding container 2183a781098a57d77f11a3cfc26d18cae759cfc6de2f0a321ef8ac9d9cb42ef8: Status 404 returned error can't find the container with id 2183a781098a57d77f11a3cfc26d18cae759cfc6de2f0a321ef8ac9d9cb42ef8
May 29 11:14:18.844918 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:18.844904    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 11:14:19.379643 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:19.379603    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6" event={"ID":"379b1eee-3ab7-4510-b811-92c0536ca24c","Type":"ContainerStarted","Data":"4fa1922b977ee4f07731173bae264fb99309985e54d0446ff3250e6b1f872b05"}
May 29 11:14:19.379643 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:19.379649    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6" event={"ID":"379b1eee-3ab7-4510-b811-92c0536ca24c","Type":"ContainerStarted","Data":"2183a781098a57d77f11a3cfc26d18cae759cfc6de2f0a321ef8ac9d9cb42ef8"}
May 29 11:14:19.379930 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:19.379770    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"
May 29 11:14:19.395717 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:19.395667    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6" podStartSLOduration=1.395651939 podStartE2EDuration="1.395651939s" podCreationTimestamp="2026-05-29 11:14:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:14:19.394532913 +0000 UTC m=+1438.271095141" watchObservedRunningTime="2026-05-29 11:14:19.395651939 +0000 UTC m=+1438.272214168"
May 29 11:14:30.384816 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:30.384785    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-2qld6"
May 29 11:14:36.638837 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.638787    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"]
May 29 11:14:36.643734 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.643714    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.646142 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.646111    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 29 11:14:36.646335 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.646209    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 11:14:36.646391 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.646375    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 11:14:36.646434 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.646391    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 29 11:14:36.646581 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.646564    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 29 11:14:36.646968 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.646946    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 29 11:14:36.647190 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.647169    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-s99zf\""
May 29 11:14:36.653533 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.653510    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"]
May 29 11:14:36.731190 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.731160    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72242c4f-c432-48ba-847a-39cc955d8f5f-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.731190 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.731195    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghf6\" (UniqueName: \"kubernetes.io/projected/72242c4f-c432-48ba-847a-39cc955d8f5f-kube-api-access-mghf6\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.731470 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.731218    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.731470 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.731264    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.731470 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.731284    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.731470 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.731341    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.832565 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.832526    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.832565 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.832569    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.832836 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.832592    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.832836 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.832625    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.832836 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.832674    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72242c4f-c432-48ba-847a-39cc955d8f5f-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.832836 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:14:36.832688    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 29 11:14:36.832836 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:14:36.832753    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 29 11:14:36.832836 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:14:36.832772    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-registry-tls podName:72242c4f-c432-48ba-847a-39cc955d8f5f nodeName:}" failed. No retries permitted until 2026-05-29 11:14:37.332748268 +0000 UTC m=+1456.209310490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-nthj5" (UID: "72242c4f-c432-48ba-847a-39cc955d8f5f") : secret "feast-simple-feast-setup-registry-tls" not found
May 29 11:14:36.832836 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:14:36.832829    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-online-tls podName:72242c4f-c432-48ba-847a-39cc955d8f5f nodeName:}" failed. No retries permitted until 2026-05-29 11:14:37.332810727 +0000 UTC m=+1456.209372946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-online-tls") pod "feast-simple-feast-setup-664fcfcc58-nthj5" (UID: "72242c4f-c432-48ba-847a-39cc955d8f5f") : secret "feast-simple-feast-setup-online-tls" not found
May 29 11:14:36.833180 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.832880    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mghf6\" (UniqueName: \"kubernetes.io/projected/72242c4f-c432-48ba-847a-39cc955d8f5f-kube-api-access-mghf6\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.833180 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.833006    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72242c4f-c432-48ba-847a-39cc955d8f5f-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.835210 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.835185    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.835345 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.835185    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:36.844321 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:36.844289    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghf6\" (UniqueName: \"kubernetes.io/projected/72242c4f-c432-48ba-847a-39cc955d8f5f-kube-api-access-mghf6\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:37.336988 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:37.336946    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:37.337188 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:37.337036    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:37.339456 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:37.339423    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:37.339567 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:37.339504    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nthj5\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:37.571993 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:37.571941    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:37.707358 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:37.707268    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"]
May 29 11:14:37.709224 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:14:37.709194    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72242c4f_c432_48ba_847a_39cc955d8f5f.slice/crio-102476ab5427e52e5842b3a78c938c18777998d378185361210291827e80b1a3 WatchSource:0}: Error finding container 102476ab5427e52e5842b3a78c938c18777998d378185361210291827e80b1a3: Status 404 returned error can't find the container with id 102476ab5427e52e5842b3a78c938c18777998d378185361210291827e80b1a3
May 29 11:14:38.438169 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:38.438127    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerStarted","Data":"5a1742324d39c61ca878d8a70d099a8e91e3eea228e644e50becfe041e1c4e9d"}
May 29 11:14:38.438169 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:38.438167    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerStarted","Data":"102476ab5427e52e5842b3a78c938c18777998d378185361210291827e80b1a3"}
May 29 11:14:42.453932 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:42.453900    2547 generic.go:358] "Generic (PLEG): container finished" podID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerID="5a1742324d39c61ca878d8a70d099a8e91e3eea228e644e50becfe041e1c4e9d" exitCode=0
May 29 11:14:42.454309 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:42.453960    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerDied","Data":"5a1742324d39c61ca878d8a70d099a8e91e3eea228e644e50becfe041e1c4e9d"}
May 29 11:14:43.458998 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:43.458958    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerStarted","Data":"10c37379756b05f7f869c5032d843f6179cc0523613ee04705e3c84dc099ce64"}
May 29 11:14:48.477640 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:48.477604    2547 generic.go:358] "Generic (PLEG): container finished" podID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerID="10c37379756b05f7f869c5032d843f6179cc0523613ee04705e3c84dc099ce64" exitCode=0
May 29 11:14:48.478049 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:48.477677    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerDied","Data":"10c37379756b05f7f869c5032d843f6179cc0523613ee04705e3c84dc099ce64"}
May 29 11:14:49.484820 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.484755    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerStarted","Data":"7c354188c8832d4a8e80b956ad4df2c6ffefee2c2a83ebb67515f4003fa53a43"}
May 29 11:14:49.485328 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.484828    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerStarted","Data":"ac4616163a2bf3799cc84cc9f778a97ba54b55d074242179c0e26b145966dd78"}
May 29 11:14:49.485328 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.484844    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerStarted","Data":"17d3649a52f0df95cd60bfe53a536933e1b17a2cdb817f5c9d3554df01f6c677"}
May 29 11:14:49.485328 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.484857    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerStarted","Data":"ebb0b68276a0bf44148229dd6ae9df8f189fcb7b67511e319cd514d86a510ae7"}
May 29 11:14:49.510210 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.510145    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podStartSLOduration=13.510124876999999 podStartE2EDuration="13.510124877s" podCreationTimestamp="2026-05-29 11:14:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:14:49.507185851 +0000 UTC m=+1468.383748080" watchObservedRunningTime="2026-05-29 11:14:49.510124877 +0000 UTC m=+1468.386687105"
May 29 11:14:49.572734 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.572695    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:49.572944 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.572744    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:49.572944 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.572758    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:49.572944 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.572770    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:49.574992 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.574948    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.39:8816: connect: connection refused"
May 29 11:14:49.575133 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.574990    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 29 11:14:49.575133 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.575076    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 29 11:14:49.575326 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:49.575294    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.39:6571: connect: connection refused"
May 29 11:14:52.572838 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:52.572778    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.39:6571: connect: connection refused"
May 29 11:14:52.572838 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:52.572807    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 29 11:14:52.573444 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:52.572778    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.39:8816: connect: connection refused"
May 29 11:14:52.573444 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:52.573298    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 29 11:14:55.573434 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.573401    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:55.573889 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.573524    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:55.573889 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.573791    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:55.574173 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.574074    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:55.574173 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.574104    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:55.574173 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.574117    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:55.574429 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.574408    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:55.574480 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.574470    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:55.574862 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.574844    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:55.578832 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:55.578805    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:56.510156 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:56.510117    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:14:56.513294 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:14:56.513274    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:15:21.748668 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:21.748637    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 11:15:21.750578 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:21.750552    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 11:15:58.599590 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:58.599556    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"]
May 29 11:15:58.600086 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:58.600000    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="registry" containerID="cri-o://ebb0b68276a0bf44148229dd6ae9df8f189fcb7b67511e319cd514d86a510ae7" gracePeriod=30
May 29 11:15:58.600086 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:58.600070    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="ui" containerID="cri-o://7c354188c8832d4a8e80b956ad4df2c6ffefee2c2a83ebb67515f4003fa53a43" gracePeriod=30
May 29 11:15:58.600217 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:58.600109    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="online" containerID="cri-o://17d3649a52f0df95cd60bfe53a536933e1b17a2cdb817f5c9d3554df01f6c677" gracePeriod=30
May 29 11:15:58.600217 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:58.600088    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="offline" containerID="cri-o://ac4616163a2bf3799cc84cc9f778a97ba54b55d074242179c0e26b145966dd78" gracePeriod=30
May 29 11:15:58.995759 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:58.995665    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"]
May 29 11:15:59.000509 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.000480    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.002690 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.002663    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-zwhqc\""
May 29 11:15:59.011026 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.010052    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"]
May 29 11:15:59.144131 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.144092    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.144313 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.144146    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.144313 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.144250    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.144313 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.144276    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sph56\" (UniqueName: \"kubernetes.io/projected/e8d3b1b4-78e3-40ee-8fee-cc1592169068-kube-api-access-sph56\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.144313 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.144297    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d3b1b4-78e3-40ee-8fee-cc1592169068-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.144469 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.144330    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.245120 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.245081    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.245120 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.245120    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sph56\" (UniqueName: \"kubernetes.io/projected/e8d3b1b4-78e3-40ee-8fee-cc1592169068-kube-api-access-sph56\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.245412 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.245140    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d3b1b4-78e3-40ee-8fee-cc1592169068-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.245412 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.245170    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.245412 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.245206    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.245412 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.245232    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.245412 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:15:59.245353    2547 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 29 11:15:59.245685 ip-10-0-142-12 kubenswrapper[2547]: E0529 11:15:59.245424    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-ui-tls podName:e8d3b1b4-78e3-40ee-8fee-cc1592169068 nodeName:}" failed. No retries permitted until 2026-05-29 11:15:59.745404875 +0000 UTC m=+1538.621967088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-6ndbt" (UID: "e8d3b1b4-78e3-40ee-8fee-cc1592169068") : secret "feast-simple-feast-setup-ui-tls" not found
May 29 11:15:59.245685 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.245625    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d3b1b4-78e3-40ee-8fee-cc1592169068-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.247818 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.247795    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.247945 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.247868    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.248011 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.247962    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.253438 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.253414    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sph56\" (UniqueName: \"kubernetes.io/projected/e8d3b1b4-78e3-40ee-8fee-cc1592169068-kube-api-access-sph56\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.710448 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.710413    2547 generic.go:358] "Generic (PLEG): container finished" podID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerID="7c354188c8832d4a8e80b956ad4df2c6ffefee2c2a83ebb67515f4003fa53a43" exitCode=0
May 29 11:15:59.710448 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.710440    2547 generic.go:358] "Generic (PLEG): container finished" podID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerID="17d3649a52f0df95cd60bfe53a536933e1b17a2cdb817f5c9d3554df01f6c677" exitCode=0
May 29 11:15:59.710860 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.710471    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerDied","Data":"7c354188c8832d4a8e80b956ad4df2c6ffefee2c2a83ebb67515f4003fa53a43"}
May 29 11:15:59.710860 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.710493    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerDied","Data":"17d3649a52f0df95cd60bfe53a536933e1b17a2cdb817f5c9d3554df01f6c677"}
May 29 11:15:59.748627 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.748587    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.750972 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.750948    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6ndbt\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:15:59.914400 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:15:59.914361    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:00.047150 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:00.047115    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"]
May 29 11:16:00.047990 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:16:00.047966    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8d3b1b4_78e3_40ee_8fee_cc1592169068.slice/crio-fc1b436df72d6c822e20936d12549f6e548318fd7cedbf3567ff98971aef3297 WatchSource:0}: Error finding container fc1b436df72d6c822e20936d12549f6e548318fd7cedbf3567ff98971aef3297: Status 404 returned error can't find the container with id fc1b436df72d6c822e20936d12549f6e548318fd7cedbf3567ff98971aef3297
May 29 11:16:00.714635 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:00.714595    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerStarted","Data":"2ee1f1c49b56055949a5b6083bea68265f6753b5bfd469357414fecde0807b7b"}
May 29 11:16:00.714635 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:00.714630    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerStarted","Data":"fc1b436df72d6c822e20936d12549f6e548318fd7cedbf3567ff98971aef3297"}
May 29 11:16:04.727459 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:04.727426    2547 generic.go:358] "Generic (PLEG): container finished" podID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerID="2ee1f1c49b56055949a5b6083bea68265f6753b5bfd469357414fecde0807b7b" exitCode=0
May 29 11:16:04.727866 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:04.727507    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerDied","Data":"2ee1f1c49b56055949a5b6083bea68265f6753b5bfd469357414fecde0807b7b"}
May 29 11:16:05.575003 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:05.574953    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 29 11:16:05.732296 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:05.732256    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerStarted","Data":"0df38e1d580c92d1d19c2f0e8678a1fe091c3692391ad71e400233230a6e5326"}
May 29 11:16:06.510641 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:06.510591    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 29 11:16:10.749611 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:10.749517    2547 generic.go:358] "Generic (PLEG): container finished" podID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerID="0df38e1d580c92d1d19c2f0e8678a1fe091c3692391ad71e400233230a6e5326" exitCode=0
May 29 11:16:10.749611 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:10.749593    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerDied","Data":"0df38e1d580c92d1d19c2f0e8678a1fe091c3692391ad71e400233230a6e5326"}
May 29 11:16:11.757221 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.757174    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerStarted","Data":"7773b4bbf67457f3af6cc12eba314c0d443dd6f0cc276e3c5120a3ca3a519f05"}
May 29 11:16:11.757221 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.757229    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerStarted","Data":"52c99551f6a51fff0cb85b6cae38c8fae3ce56b05f99f68f08f009fc6168402e"}
May 29 11:16:11.757798 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.757264    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerStarted","Data":"958db44b9610ecf6b9a1dd6795f25d6376d5a0486a41f538a34637d4c15c8bd3"}
May 29 11:16:11.757798 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.757278    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerStarted","Data":"9448e374ae3926a0df2ee3167d40a2969661e065ae5537798432c9b543b02f99"}
May 29 11:16:11.782631 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.782558    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podStartSLOduration=13.782538514 podStartE2EDuration="13.782538514s" podCreationTimestamp="2026-05-29 11:15:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:16:11.779676701 +0000 UTC m=+1550.656238931" watchObservedRunningTime="2026-05-29 11:16:11.782538514 +0000 UTC m=+1550.659100745"
May 29 11:16:11.915225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.915026    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:11.915225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.915088    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:11.915225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.915189    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:11.915545 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.915260    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:11.917141 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.917036    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused"
May 29 11:16:11.917141 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.917079    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.40:6571: connect: connection refused"
May 29 11:16:11.917141 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.917085    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused"
May 29 11:16:11.917449 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:11.917121    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.40:8816: connect: connection refused"
May 29 11:16:14.915407 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:14.915364    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.40:6571: connect: connection refused"
May 29 11:16:14.916000 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:14.915361    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.40:8816: connect: connection refused"
May 29 11:16:14.916197 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:14.915361    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused"
May 29 11:16:14.916370 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:14.915374    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused"
May 29 11:16:15.574376 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:15.574323    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 29 11:16:16.510913 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:16.510864    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 29 11:16:17.916133 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:17.916089    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:17.916638 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:17.916622    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:17.916705 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:17.916656    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:17.916781 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:17.916766    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:17.916885 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:17.916866    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:17.920842 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:17.920820    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:18.784770 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:18.784738    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:18.784770 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:18.784776    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:18.785032 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:18.784788    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:18.785032 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:18.785017    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:18.785324 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:18.785306    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:18.787594 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:18.787572    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:16:25.574888 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:25.574830    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused"
May 29 11:16:25.577441 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:25.574995    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:16:26.510516 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:26.510475    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 29 11:16:26.510689 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:26.510599    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:16:28.816835 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:28.816805    2547 generic.go:358] "Generic (PLEG): container finished" podID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerID="ac4616163a2bf3799cc84cc9f778a97ba54b55d074242179c0e26b145966dd78" exitCode=137
May 29 11:16:28.816835 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:28.816829    2547 generic.go:358] "Generic (PLEG): container finished" podID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerID="ebb0b68276a0bf44148229dd6ae9df8f189fcb7b67511e319cd514d86a510ae7" exitCode=137
May 29 11:16:28.817274 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:28.816868    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerDied","Data":"ac4616163a2bf3799cc84cc9f778a97ba54b55d074242179c0e26b145966dd78"}
May 29 11:16:28.817274 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:28.816899    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerDied","Data":"ebb0b68276a0bf44148229dd6ae9df8f189fcb7b67511e319cd514d86a510ae7"}
May 29 11:16:29.253517 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.253489    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:16:29.407561 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.407470    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-offline-tls\") pod \"72242c4f-c432-48ba-847a-39cc955d8f5f\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") "
May 29 11:16:29.407561 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.407527    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72242c4f-c432-48ba-847a-39cc955d8f5f-feast-data\") pod \"72242c4f-c432-48ba-847a-39cc955d8f5f\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") "
May 29 11:16:29.407812 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.407591    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mghf6\" (UniqueName: \"kubernetes.io/projected/72242c4f-c432-48ba-847a-39cc955d8f5f-kube-api-access-mghf6\") pod \"72242c4f-c432-48ba-847a-39cc955d8f5f\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") "
May 29 11:16:29.407812 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.407613    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-online-tls\") pod \"72242c4f-c432-48ba-847a-39cc955d8f5f\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") "
May 29 11:16:29.407812 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.407629    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-ui-tls\") pod \"72242c4f-c432-48ba-847a-39cc955d8f5f\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") "
May 29 11:16:29.407812 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.407657    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-registry-tls\") pod \"72242c4f-c432-48ba-847a-39cc955d8f5f\" (UID: \"72242c4f-c432-48ba-847a-39cc955d8f5f\") "
May 29 11:16:29.408148 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.408119    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72242c4f-c432-48ba-847a-39cc955d8f5f-feast-data" (OuterVolumeSpecName: "feast-data") pod "72242c4f-c432-48ba-847a-39cc955d8f5f" (UID: "72242c4f-c432-48ba-847a-39cc955d8f5f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:16:29.409954 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.409924    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "72242c4f-c432-48ba-847a-39cc955d8f5f" (UID: "72242c4f-c432-48ba-847a-39cc955d8f5f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:16:29.410203 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.410175    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-online-tls" (OuterVolumeSpecName: "online-tls") pod "72242c4f-c432-48ba-847a-39cc955d8f5f" (UID: "72242c4f-c432-48ba-847a-39cc955d8f5f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:16:29.410203 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.410194    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "72242c4f-c432-48ba-847a-39cc955d8f5f" (UID: "72242c4f-c432-48ba-847a-39cc955d8f5f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:16:29.410363 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.410266    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72242c4f-c432-48ba-847a-39cc955d8f5f-kube-api-access-mghf6" (OuterVolumeSpecName: "kube-api-access-mghf6") pod "72242c4f-c432-48ba-847a-39cc955d8f5f" (UID: "72242c4f-c432-48ba-847a-39cc955d8f5f"). InnerVolumeSpecName "kube-api-access-mghf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:16:29.410363 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.410270    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "72242c4f-c432-48ba-847a-39cc955d8f5f" (UID: "72242c4f-c432-48ba-847a-39cc955d8f5f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:16:29.508377 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.508346    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghf6\" (UniqueName: \"kubernetes.io/projected/72242c4f-c432-48ba-847a-39cc955d8f5f-kube-api-access-mghf6\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:16:29.508377 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.508372    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:16:29.508377 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.508383    2547 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-ui-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:16:29.508377 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.508391    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:16:29.508377 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.508400    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/72242c4f-c432-48ba-847a-39cc955d8f5f-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:16:29.508743 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.508407    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72242c4f-c432-48ba-847a-39cc955d8f5f-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:16:29.822037 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.822001    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5" event={"ID":"72242c4f-c432-48ba-847a-39cc955d8f5f","Type":"ContainerDied","Data":"102476ab5427e52e5842b3a78c938c18777998d378185361210291827e80b1a3"}
May 29 11:16:29.822037 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.822027    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"
May 29 11:16:29.822503 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.822057    2547 scope.go:117] "RemoveContainer" containerID="7c354188c8832d4a8e80b956ad4df2c6ffefee2c2a83ebb67515f4003fa53a43"
May 29 11:16:29.830948 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.830933    2547 scope.go:117] "RemoveContainer" containerID="ac4616163a2bf3799cc84cc9f778a97ba54b55d074242179c0e26b145966dd78"
May 29 11:16:29.838047 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.838033    2547 scope.go:117] "RemoveContainer" containerID="17d3649a52f0df95cd60bfe53a536933e1b17a2cdb817f5c9d3554df01f6c677"
May 29 11:16:29.844122 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.844101    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"]
May 29 11:16:29.846632 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.846605    2547 scope.go:117] "RemoveContainer" containerID="ebb0b68276a0bf44148229dd6ae9df8f189fcb7b67511e319cd514d86a510ae7"
May 29 11:16:29.849916 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.849889    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nthj5"]
May 29 11:16:29.856311 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.856292    2547 scope.go:117] "RemoveContainer" containerID="10c37379756b05f7f869c5032d843f6179cc0523613ee04705e3c84dc099ce64"
May 29 11:16:29.863284 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:29.863259    2547 scope.go:117] "RemoveContainer" containerID="5a1742324d39c61ca878d8a70d099a8e91e3eea228e644e50becfe041e1c4e9d"
May 29 11:16:31.717324 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:16:31.717291    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" path="/var/lib/kubelet/pods/72242c4f-c432-48ba-847a-39cc955d8f5f/volumes"
May 29 11:17:20.530419 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530334    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"]
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530656    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="ui"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530666    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="ui"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530679    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="registry"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530686    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="registry"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530695    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="offline"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530701    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="offline"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530709    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="feast-apply"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530714    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="feast-apply"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530726    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="online"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530731    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="online"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530743    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="feast-init"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530749    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="feast-init"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530806    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="registry"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530815    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="ui"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530824    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="offline"
May 29 11:17:20.530921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.530830    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="72242c4f-c432-48ba-847a-39cc955d8f5f" containerName="online"
May 29 11:17:20.534128 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.534107    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.541301 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.541281    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 29 11:17:20.541386 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.541286    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 29 11:17:20.541656 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.541642    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 29 11:17:20.541882 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.541867    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 29 11:17:20.542781 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.542759    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-whddp\""
May 29 11:17:20.542837 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.542781    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 29 11:17:20.543123 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.543110    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 29 11:17:20.553053 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.553029    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"]
May 29 11:17:20.599937 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.599903    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/024e8c54-fb10-46f8-921e-93b0b8469e9e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.599937 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.599939    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.600128 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.599974    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.600128 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.600014    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/024e8c54-fb10-46f8-921e-93b0b8469e9e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.600128 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.600037    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm677\" (UniqueName: \"kubernetes.io/projected/024e8c54-fb10-46f8-921e-93b0b8469e9e-kube-api-access-lm677\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.600227 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.600137    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.701106 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.701075    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.701322 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.701130    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/024e8c54-fb10-46f8-921e-93b0b8469e9e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.701322 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.701156    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.701322 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.701189    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.701322 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.701219    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/024e8c54-fb10-46f8-921e-93b0b8469e9e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.701322 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.701273    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lm677\" (UniqueName: \"kubernetes.io/projected/024e8c54-fb10-46f8-921e-93b0b8469e9e-kube-api-access-lm677\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.701706 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.701676    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/024e8c54-fb10-46f8-921e-93b0b8469e9e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.701877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.701854    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/024e8c54-fb10-46f8-921e-93b0b8469e9e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.703747 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.703722    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.703859 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.703779    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.703859 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.703798    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.713115 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.713093    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm677\" (UniqueName: \"kubernetes.io/projected/024e8c54-fb10-46f8-921e-93b0b8469e9e-kube-api-access-lm677\") pod \"feast-simple-feast-remote-setup-5cff94746b-qscml\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.843543 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.843456    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:20.984277 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:20.984223    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"]
May 29 11:17:20.987320 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:17:20.987289    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod024e8c54_fb10_46f8_921e_93b0b8469e9e.slice/crio-031870e714c3e2db9f314078b1c1f74fb98ce340512be4b5883bd208277685bd WatchSource:0}: Error finding container 031870e714c3e2db9f314078b1c1f74fb98ce340512be4b5883bd208277685bd: Status 404 returned error can't find the container with id 031870e714c3e2db9f314078b1c1f74fb98ce340512be4b5883bd208277685bd
May 29 11:17:21.976671 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:21.976632    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerStarted","Data":"f19ac1a9f725787d44c41d85338aad01ee56860c91ded54c8fbe1167c5e6605d"}
May 29 11:17:21.976671 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:21.976676    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerStarted","Data":"031870e714c3e2db9f314078b1c1f74fb98ce340512be4b5883bd208277685bd"}
May 29 11:17:25.989411 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:25.989319    2547 generic.go:358] "Generic (PLEG): container finished" podID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerID="f19ac1a9f725787d44c41d85338aad01ee56860c91ded54c8fbe1167c5e6605d" exitCode=0
May 29 11:17:25.989411 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:25.989394    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerDied","Data":"f19ac1a9f725787d44c41d85338aad01ee56860c91ded54c8fbe1167c5e6605d"}
May 29 11:17:26.994693 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:26.994646    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerStarted","Data":"c45c8efc9deb796b1d4bd5c6d4290dae8fa4bff9b49f8f1775b8bde641744e34"}
May 29 11:17:32.013143 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:32.013099    2547 generic.go:358] "Generic (PLEG): container finished" podID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerID="c45c8efc9deb796b1d4bd5c6d4290dae8fa4bff9b49f8f1775b8bde641744e34" exitCode=0
May 29 11:17:32.013143 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:32.013146    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerDied","Data":"c45c8efc9deb796b1d4bd5c6d4290dae8fa4bff9b49f8f1775b8bde641744e34"}
May 29 11:17:33.028724 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:33.028683    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerStarted","Data":"d3e8cb7ec7219ab0691c1e3c6752d583658b899b6ba836cfe7b535eaeed4cd01"}
May 29 11:17:33.028724 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:33.028726    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerStarted","Data":"2b01393fcb5f0fc7e11813f5a4be3160c3baf9dfa17973057477228e68f11963"}
May 29 11:17:33.029341 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:33.028736    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerStarted","Data":"8ec5af0d601c6c238378e3a582ed8a7da847e874ad97b2bdc6a23e765183ef0b"}
May 29 11:17:33.076921 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:33.076853    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podStartSLOduration=13.076830916 podStartE2EDuration="13.076830916s" podCreationTimestamp="2026-05-29 11:17:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:17:33.074327124 +0000 UTC m=+1631.950889358" watchObservedRunningTime="2026-05-29 11:17:33.076830916 +0000 UTC m=+1631.953393144"
May 29 11:17:35.844191 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:35.844150    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:35.844796 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:35.844765    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:35.844796 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:35.844794    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:35.845955 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:35.845921    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.41:8816: connect: connection refused"
May 29 11:17:35.845955 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:35.845938    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused"
May 29 11:17:35.846118 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:35.846005    2547 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused"
May 29 11:17:38.845618 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:38.845590    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:38.846498 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:38.845702    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:38.846498 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:38.846093    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:38.846498 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:38.846121    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:38.846498 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:38.846374    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:38.846765 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:38.846511    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:38.850152 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:38.850124    2547 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:39.049951 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:39.049923    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:17:39.052859 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:17:39.052831    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:18:22.430219 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:22.430183    2547 scope.go:117] "RemoveContainer" containerID="9e6e777b454ee3bf40ce80d2cf7eac3a865ae52e5159452dd48eac948c829fcc"
May 29 11:18:22.438935 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:22.438908    2547 scope.go:117] "RemoveContainer" containerID="c7ece10c2c780b7c2c83813391c3d8d7621cb0025069c801bc3cbfca0a6cdf87"
May 29 11:18:22.447325 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:22.447218    2547 scope.go:117] "RemoveContainer" containerID="14144a4c9e2fe9bfb6d91bac91279a43882e15b28410b27f0bd4fe3c34250811"
May 29 11:18:22.454929 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:22.454909    2547 scope.go:117] "RemoveContainer" containerID="7699b1a19f73afebcdb644f6e5f5875d26c22156f5dc8cee006061e359b7f22c"
May 29 11:18:22.464667 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:22.464643    2547 scope.go:117] "RemoveContainer" containerID="c246bf4eb8bf3f8ab16805a3dea5c341d3661dc26fa11e0a271a3068cca247b0"
May 29 11:18:41.720959 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:41.720625    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"]
May 29 11:18:41.721529 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:41.721366    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="online" containerID="cri-o://8ec5af0d601c6c238378e3a582ed8a7da847e874ad97b2bdc6a23e765183ef0b" gracePeriod=30
May 29 11:18:41.721529 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:41.721503    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="ui" containerID="cri-o://d3e8cb7ec7219ab0691c1e3c6752d583658b899b6ba836cfe7b535eaeed4cd01" gracePeriod=30
May 29 11:18:41.721657 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:41.721549    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="offline" containerID="cri-o://2b01393fcb5f0fc7e11813f5a4be3160c3baf9dfa17973057477228e68f11963" gracePeriod=30
May 29 11:18:41.904529 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:41.904485    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"]
May 29 11:18:41.904888 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:41.904836    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="registry" containerID="cri-o://9448e374ae3926a0df2ee3167d40a2969661e065ae5537798432c9b543b02f99" gracePeriod=30
May 29 11:18:41.905029 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:41.904884    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="online" containerID="cri-o://958db44b9610ecf6b9a1dd6795f25d6376d5a0486a41f538a34637d4c15c8bd3" gracePeriod=30
May 29 11:18:41.905029 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:41.904887    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="ui" containerID="cri-o://7773b4bbf67457f3af6cc12eba314c0d443dd6f0cc276e3c5120a3ca3a519f05" gracePeriod=30
May 29 11:18:41.905029 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:41.904884    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="offline" containerID="cri-o://52c99551f6a51fff0cb85b6cae38c8fae3ce56b05f99f68f08f009fc6168402e" gracePeriod=30
May 29 11:18:43.250852 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:43.250812    2547 generic.go:358] "Generic (PLEG): container finished" podID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerID="7773b4bbf67457f3af6cc12eba314c0d443dd6f0cc276e3c5120a3ca3a519f05" exitCode=0
May 29 11:18:43.250852 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:43.250843    2547 generic.go:358] "Generic (PLEG): container finished" podID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerID="958db44b9610ecf6b9a1dd6795f25d6376d5a0486a41f538a34637d4c15c8bd3" exitCode=0
May 29 11:18:43.251311 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:43.250886    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerDied","Data":"7773b4bbf67457f3af6cc12eba314c0d443dd6f0cc276e3c5120a3ca3a519f05"}
May 29 11:18:43.251311 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:43.250944    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerDied","Data":"958db44b9610ecf6b9a1dd6795f25d6376d5a0486a41f538a34637d4c15c8bd3"}
May 29 11:18:43.253155 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:43.253128    2547 generic.go:358] "Generic (PLEG): container finished" podID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerID="d3e8cb7ec7219ab0691c1e3c6752d583658b899b6ba836cfe7b535eaeed4cd01" exitCode=0
May 29 11:18:43.253155 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:43.253146    2547 generic.go:358] "Generic (PLEG): container finished" podID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerID="8ec5af0d601c6c238378e3a582ed8a7da847e874ad97b2bdc6a23e765183ef0b" exitCode=0
May 29 11:18:43.253371 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:43.253199    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerDied","Data":"d3e8cb7ec7219ab0691c1e3c6752d583658b899b6ba836cfe7b535eaeed4cd01"}
May 29 11:18:43.253371 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:43.253251    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerDied","Data":"8ec5af0d601c6c238378e3a582ed8a7da847e874ad97b2bdc6a23e765183ef0b"}
May 29 11:18:48.785052 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:48.785014    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused"
May 29 11:18:48.785483 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:48.785047    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused"
May 29 11:18:48.847105 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:48.847068    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused"
May 29 11:18:49.051081 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:49.050990    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused"
May 29 11:18:58.785150 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:58.785098    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused"
May 29 11:18:58.785705 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:58.785109    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused"
May 29 11:18:58.846426 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:58.846388    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused"
May 29 11:18:59.050514 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:18:59.050410    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused"
May 29 11:19:08.785810 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:08.785756    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused"
May 29 11:19:08.786328 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:08.785765    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused"
May 29 11:19:08.786328 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:08.785909    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:19:08.786328 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:08.786097    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:19:08.846311 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:08.846267    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused"
May 29 11:19:08.846489 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:08.846414    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:19:09.050593 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:09.050493    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused"
May 29 11:19:09.050753 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:09.050624    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:19:12.346688 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.346643    2547 generic.go:358] "Generic (PLEG): container finished" podID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerID="52c99551f6a51fff0cb85b6cae38c8fae3ce56b05f99f68f08f009fc6168402e" exitCode=137
May 29 11:19:12.346688 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.346673    2547 generic.go:358] "Generic (PLEG): container finished" podID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerID="9448e374ae3926a0df2ee3167d40a2969661e065ae5537798432c9b543b02f99" exitCode=137
May 29 11:19:12.347290 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.346709    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerDied","Data":"52c99551f6a51fff0cb85b6cae38c8fae3ce56b05f99f68f08f009fc6168402e"}
May 29 11:19:12.347290 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.346753    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerDied","Data":"9448e374ae3926a0df2ee3167d40a2969661e065ae5537798432c9b543b02f99"}
May 29 11:19:12.349290 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.349266    2547 generic.go:358] "Generic (PLEG): container finished" podID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerID="2b01393fcb5f0fc7e11813f5a4be3160c3baf9dfa17973057477228e68f11963" exitCode=137
May 29 11:19:12.349419 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.349324    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerDied","Data":"2b01393fcb5f0fc7e11813f5a4be3160c3baf9dfa17973057477228e68f11963"}
May 29 11:19:12.375762 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.375734    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:19:12.425663 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.425630    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/024e8c54-fb10-46f8-921e-93b0b8469e9e-feast-data\") pod \"024e8c54-fb10-46f8-921e-93b0b8469e9e\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") "
May 29 11:19:12.425816 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.425685    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm677\" (UniqueName: \"kubernetes.io/projected/024e8c54-fb10-46f8-921e-93b0b8469e9e-kube-api-access-lm677\") pod \"024e8c54-fb10-46f8-921e-93b0b8469e9e\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") "
May 29 11:19:12.425816 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.425720    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-ui-tls\") pod \"024e8c54-fb10-46f8-921e-93b0b8469e9e\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") "
May 29 11:19:12.425816 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.425774    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-offline-tls\") pod \"024e8c54-fb10-46f8-921e-93b0b8469e9e\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") "
May 29 11:19:12.426060 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.425821    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/024e8c54-fb10-46f8-921e-93b0b8469e9e-registry-tls\") pod \"024e8c54-fb10-46f8-921e-93b0b8469e9e\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") "
May 29 11:19:12.426060 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.425872    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-online-tls\") pod \"024e8c54-fb10-46f8-921e-93b0b8469e9e\" (UID: \"024e8c54-fb10-46f8-921e-93b0b8469e9e\") "
May 29 11:19:12.426175 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.426097    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e8c54-fb10-46f8-921e-93b0b8469e9e-feast-data" (OuterVolumeSpecName: "feast-data") pod "024e8c54-fb10-46f8-921e-93b0b8469e9e" (UID: "024e8c54-fb10-46f8-921e-93b0b8469e9e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:19:12.426662 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.426614    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024e8c54-fb10-46f8-921e-93b0b8469e9e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "024e8c54-fb10-46f8-921e-93b0b8469e9e" (UID: "024e8c54-fb10-46f8-921e-93b0b8469e9e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 11:19:12.428748 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.428723    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "024e8c54-fb10-46f8-921e-93b0b8469e9e" (UID: "024e8c54-fb10-46f8-921e-93b0b8469e9e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:19:12.428844 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.428798    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-online-tls" (OuterVolumeSpecName: "online-tls") pod "024e8c54-fb10-46f8-921e-93b0b8469e9e" (UID: "024e8c54-fb10-46f8-921e-93b0b8469e9e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:19:12.428915 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.428895    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "024e8c54-fb10-46f8-921e-93b0b8469e9e" (UID: "024e8c54-fb10-46f8-921e-93b0b8469e9e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:19:12.429002 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.428983    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024e8c54-fb10-46f8-921e-93b0b8469e9e-kube-api-access-lm677" (OuterVolumeSpecName: "kube-api-access-lm677") pod "024e8c54-fb10-46f8-921e-93b0b8469e9e" (UID: "024e8c54-fb10-46f8-921e-93b0b8469e9e"). InnerVolumeSpecName "kube-api-access-lm677". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:19:12.526981 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.526948    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/024e8c54-fb10-46f8-921e-93b0b8469e9e-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.526981 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.526979    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.526981 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.526989    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/024e8c54-fb10-46f8-921e-93b0b8469e9e-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.527304 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.526998    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm677\" (UniqueName: \"kubernetes.io/projected/024e8c54-fb10-46f8-921e-93b0b8469e9e-kube-api-access-lm677\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.527304 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.527007    2547 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-ui-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.527304 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.527017    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/024e8c54-fb10-46f8-921e-93b0b8469e9e-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.555271 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.555216    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:19:12.628098 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.628065    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d3b1b4-78e3-40ee-8fee-cc1592169068-feast-data\") pod \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") "
May 29 11:19:12.628319 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.628131    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-offline-tls\") pod \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") "
May 29 11:19:12.628319 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.628167    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sph56\" (UniqueName: \"kubernetes.io/projected/e8d3b1b4-78e3-40ee-8fee-cc1592169068-kube-api-access-sph56\") pod \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") "
May 29 11:19:12.628319 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.628188    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-registry-tls\") pod \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") "
May 29 11:19:12.628319 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.628215    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-ui-tls\") pod \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") "
May 29 11:19:12.628319 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.628256    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-online-tls\") pod \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\" (UID: \"e8d3b1b4-78e3-40ee-8fee-cc1592169068\") "
May 29 11:19:12.628730 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.628703    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d3b1b4-78e3-40ee-8fee-cc1592169068-feast-data" (OuterVolumeSpecName: "feast-data") pod "e8d3b1b4-78e3-40ee-8fee-cc1592169068" (UID: "e8d3b1b4-78e3-40ee-8fee-cc1592169068"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:19:12.630442 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.630408    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e8d3b1b4-78e3-40ee-8fee-cc1592169068" (UID: "e8d3b1b4-78e3-40ee-8fee-cc1592169068"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:19:12.630576 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.630432    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e8d3b1b4-78e3-40ee-8fee-cc1592169068" (UID: "e8d3b1b4-78e3-40ee-8fee-cc1592169068"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:19:12.630576 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.630527    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d3b1b4-78e3-40ee-8fee-cc1592169068-kube-api-access-sph56" (OuterVolumeSpecName: "kube-api-access-sph56") pod "e8d3b1b4-78e3-40ee-8fee-cc1592169068" (UID: "e8d3b1b4-78e3-40ee-8fee-cc1592169068"). InnerVolumeSpecName "kube-api-access-sph56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:19:12.630758 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.630739    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-online-tls" (OuterVolumeSpecName: "online-tls") pod "e8d3b1b4-78e3-40ee-8fee-cc1592169068" (UID: "e8d3b1b4-78e3-40ee-8fee-cc1592169068"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:19:12.630836 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.630808    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e8d3b1b4-78e3-40ee-8fee-cc1592169068" (UID: "e8d3b1b4-78e3-40ee-8fee-cc1592169068"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 11:19:12.729507 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.729463    2547 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-offline-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.729507 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.729507    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sph56\" (UniqueName: \"kubernetes.io/projected/e8d3b1b4-78e3-40ee-8fee-cc1592169068-kube-api-access-sph56\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.729714 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.729521    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-registry-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.729714 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.729536    2547 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-ui-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.729714 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.729548    2547 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8d3b1b4-78e3-40ee-8fee-cc1592169068-online-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:12.729714 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:12.729560    2547 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d3b1b4-78e3-40ee-8fee-cc1592169068-feast-data\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:13.354766 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.354724    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt" event={"ID":"e8d3b1b4-78e3-40ee-8fee-cc1592169068","Type":"ContainerDied","Data":"fc1b436df72d6c822e20936d12549f6e548318fd7cedbf3567ff98971aef3297"}
May 29 11:19:13.354766 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.354759    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"
May 29 11:19:13.354766 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.354782    2547 scope.go:117] "RemoveContainer" containerID="7773b4bbf67457f3af6cc12eba314c0d443dd6f0cc276e3c5120a3ca3a519f05"
May 29 11:19:13.357226 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.357202    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml" event={"ID":"024e8c54-fb10-46f8-921e-93b0b8469e9e","Type":"ContainerDied","Data":"031870e714c3e2db9f314078b1c1f74fb98ce340512be4b5883bd208277685bd"}
May 29 11:19:13.357377 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.357340    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"
May 29 11:19:13.364814 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.364795    2547 scope.go:117] "RemoveContainer" containerID="52c99551f6a51fff0cb85b6cae38c8fae3ce56b05f99f68f08f009fc6168402e"
May 29 11:19:13.372795 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.372771    2547 scope.go:117] "RemoveContainer" containerID="958db44b9610ecf6b9a1dd6795f25d6376d5a0486a41f538a34637d4c15c8bd3"
May 29 11:19:13.379886 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.379866    2547 scope.go:117] "RemoveContainer" containerID="9448e374ae3926a0df2ee3167d40a2969661e065ae5537798432c9b543b02f99"
May 29 11:19:13.385071 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.385045    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"]
May 29 11:19:13.388340 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.388322    2547 scope.go:117] "RemoveContainer" containerID="0df38e1d580c92d1d19c2f0e8678a1fe091c3692391ad71e400233230a6e5326"
May 29 11:19:13.389139 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.389116    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6ndbt"]
May 29 11:19:13.396114 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.396095    2547 scope.go:117] "RemoveContainer" containerID="2ee1f1c49b56055949a5b6083bea68265f6753b5bfd469357414fecde0807b7b"
May 29 11:19:13.404028 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.403986    2547 scope.go:117] "RemoveContainer" containerID="d3e8cb7ec7219ab0691c1e3c6752d583658b899b6ba836cfe7b535eaeed4cd01"
May 29 11:19:13.404161 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.403989    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"]
May 29 11:19:13.406041 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.406016    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qscml"]
May 29 11:19:13.411801 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.411782    2547 scope.go:117] "RemoveContainer" containerID="2b01393fcb5f0fc7e11813f5a4be3160c3baf9dfa17973057477228e68f11963"
May 29 11:19:13.419499 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.419479    2547 scope.go:117] "RemoveContainer" containerID="8ec5af0d601c6c238378e3a582ed8a7da847e874ad97b2bdc6a23e765183ef0b"
May 29 11:19:13.428908 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.428883    2547 scope.go:117] "RemoveContainer" containerID="c45c8efc9deb796b1d4bd5c6d4290dae8fa4bff9b49f8f1775b8bde641744e34"
May 29 11:19:13.436720 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.436697    2547 scope.go:117] "RemoveContainer" containerID="f19ac1a9f725787d44c41d85338aad01ee56860c91ded54c8fbe1167c5e6605d"
May 29 11:19:13.717497 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.717417    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" path="/var/lib/kubelet/pods/024e8c54-fb10-46f8-921e-93b0b8469e9e/volumes"
May 29 11:19:13.717971 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:13.717958    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" path="/var/lib/kubelet/pods/e8d3b1b4-78e3-40ee-8fee-cc1592169068/volumes"
May 29 11:19:32.506313 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506275    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-56jlj/must-gather-dqmmn"]
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506580    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="offline"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506590    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="offline"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506601    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="registry"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506607    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="registry"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506623    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="feast-apply"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506630    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="feast-apply"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506635    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="offline"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506640    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="offline"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506648    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="feast-init"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506653    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="feast-init"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506661    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="ui"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506665    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="ui"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506673    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="online"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506678    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="online"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506684    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="online"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506688    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="online"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506695    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="ui"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506699    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="ui"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506706    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="feast-init"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506711    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="feast-init"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506719    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="feast-apply"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506724    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="feast-apply"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506777    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="offline"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506786    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="online"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506793    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="offline"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506799    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="online"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506804    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="ui"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506812    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="024e8c54-fb10-46f8-921e-93b0b8469e9e" containerName="ui"
May 29 11:19:32.506877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.506818    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8d3b1b4-78e3-40ee-8fee-cc1592169068" containerName="registry"
May 29 11:19:32.511225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.511209    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:32.515901 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.515881    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-56jlj\"/\"kube-root-ca.crt\""
May 29 11:19:32.518846 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.518816    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-56jlj\"/\"default-dockercfg-jpn82\""
May 29 11:19:32.519484 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.519466    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-56jlj\"/\"openshift-service-ca.crt\""
May 29 11:19:32.523712 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.523688    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-56jlj/must-gather-dqmmn"]
May 29 11:19:32.586679 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.586644    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-must-gather-output\") pod \"must-gather-dqmmn\" (UID: \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\") " pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:32.586881 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.586705    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trwrm\" (UniqueName: \"kubernetes.io/projected/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-kube-api-access-trwrm\") pod \"must-gather-dqmmn\" (UID: \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\") " pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:32.687297 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.687233    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-must-gather-output\") pod \"must-gather-dqmmn\" (UID: \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\") " pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:32.687470 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.687316    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-trwrm\" (UniqueName: \"kubernetes.io/projected/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-kube-api-access-trwrm\") pod \"must-gather-dqmmn\" (UID: \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\") " pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:32.687541 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.687523    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-must-gather-output\") pod \"must-gather-dqmmn\" (UID: \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\") " pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:32.696530 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.696511    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trwrm\" (UniqueName: \"kubernetes.io/projected/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-kube-api-access-trwrm\") pod \"must-gather-dqmmn\" (UID: \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\") " pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:32.834606 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.834563    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:32.961231 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.961198    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-56jlj/must-gather-dqmmn"]
May 29 11:19:32.964669 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:19:32.964641    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda05e3bd1_8c3e_45ef_9f26_e4b3b1d8f94d.slice/crio-483f6d013f812a1cc84649bbf09340d25a00631f308ba5779f5c3b0d889ddebc WatchSource:0}: Error finding container 483f6d013f812a1cc84649bbf09340d25a00631f308ba5779f5c3b0d889ddebc: Status 404 returned error can't find the container with id 483f6d013f812a1cc84649bbf09340d25a00631f308ba5779f5c3b0d889ddebc
May 29 11:19:32.966693 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:32.966678    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 11:19:33.421808 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:33.421770    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56jlj/must-gather-dqmmn" event={"ID":"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d","Type":"ContainerStarted","Data":"483f6d013f812a1cc84649bbf09340d25a00631f308ba5779f5c3b0d889ddebc"}
May 29 11:19:39.444832 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:39.444443    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56jlj/must-gather-dqmmn" event={"ID":"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d","Type":"ContainerStarted","Data":"e02e9e5fdc464c99aa43ced7c30ae0e8726ec21121b37745562ef1c10919a557"}
May 29 11:19:39.444832 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:39.444492    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56jlj/must-gather-dqmmn" event={"ID":"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d","Type":"ContainerStarted","Data":"d4f245bb484cebbfa0e55f03554c92ec96f9fd593912be2db85496811765b4f6"}
May 29 11:19:39.463843 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:39.463792    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-56jlj/must-gather-dqmmn" podStartSLOduration=1.38022549 podStartE2EDuration="7.463777646s" podCreationTimestamp="2026-05-29 11:19:32 +0000 UTC" firstStartedPulling="2026-05-29 11:19:32.966825161 +0000 UTC m=+1751.843387366" lastFinishedPulling="2026-05-29 11:19:39.05037731 +0000 UTC m=+1757.926939522" observedRunningTime="2026-05-29 11:19:39.461611321 +0000 UTC m=+1758.338173549" watchObservedRunningTime="2026-05-29 11:19:39.463777646 +0000 UTC m=+1758.340339873"
May 29 11:19:49.478639 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:49.478605    2547 generic.go:358] "Generic (PLEG): container finished" podID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" containerID="d4f245bb484cebbfa0e55f03554c92ec96f9fd593912be2db85496811765b4f6" exitCode=0
May 29 11:19:49.479111 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:49.478663    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56jlj/must-gather-dqmmn" event={"ID":"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d","Type":"ContainerDied","Data":"d4f245bb484cebbfa0e55f03554c92ec96f9fd593912be2db85496811765b4f6"}
May 29 11:19:49.479111 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:49.479010    2547 scope.go:117] "RemoveContainer" containerID="d4f245bb484cebbfa0e55f03554c92ec96f9fd593912be2db85496811765b4f6"
May 29 11:19:49.822821 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:49.822789    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-56jlj_must-gather-dqmmn_a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d/gather/0.log"
May 29 11:19:50.335122 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.335085    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bcqz5/must-gather-rjtdz"]
May 29 11:19:50.338489 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.338464    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bcqz5/must-gather-rjtdz"
May 29 11:19:50.341293 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.341231    2547 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bcqz5\"/\"default-dockercfg-46wcx\""
May 29 11:19:50.341537 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.341517    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bcqz5\"/\"openshift-service-ca.crt\""
May 29 11:19:50.342061 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.342044    2547 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bcqz5\"/\"kube-root-ca.crt\""
May 29 11:19:50.346816 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.346789    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bcqz5/must-gather-rjtdz"]
May 29 11:19:50.453470 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.453429    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/692ca66b-97c3-482e-a5dc-e9bcd77774e5-must-gather-output\") pod \"must-gather-rjtdz\" (UID: \"692ca66b-97c3-482e-a5dc-e9bcd77774e5\") " pod="openshift-must-gather-bcqz5/must-gather-rjtdz"
May 29 11:19:50.453673 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.453509    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slcxl\" (UniqueName: \"kubernetes.io/projected/692ca66b-97c3-482e-a5dc-e9bcd77774e5-kube-api-access-slcxl\") pod \"must-gather-rjtdz\" (UID: \"692ca66b-97c3-482e-a5dc-e9bcd77774e5\") " pod="openshift-must-gather-bcqz5/must-gather-rjtdz"
May 29 11:19:50.554663 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.554630    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/692ca66b-97c3-482e-a5dc-e9bcd77774e5-must-gather-output\") pod \"must-gather-rjtdz\" (UID: \"692ca66b-97c3-482e-a5dc-e9bcd77774e5\") " pod="openshift-must-gather-bcqz5/must-gather-rjtdz"
May 29 11:19:50.555066 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.554693    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slcxl\" (UniqueName: \"kubernetes.io/projected/692ca66b-97c3-482e-a5dc-e9bcd77774e5-kube-api-access-slcxl\") pod \"must-gather-rjtdz\" (UID: \"692ca66b-97c3-482e-a5dc-e9bcd77774e5\") " pod="openshift-must-gather-bcqz5/must-gather-rjtdz"
May 29 11:19:50.555066 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.554943    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/692ca66b-97c3-482e-a5dc-e9bcd77774e5-must-gather-output\") pod \"must-gather-rjtdz\" (UID: \"692ca66b-97c3-482e-a5dc-e9bcd77774e5\") " pod="openshift-must-gather-bcqz5/must-gather-rjtdz"
May 29 11:19:50.563143 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.563106    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slcxl\" (UniqueName: \"kubernetes.io/projected/692ca66b-97c3-482e-a5dc-e9bcd77774e5-kube-api-access-slcxl\") pod \"must-gather-rjtdz\" (UID: \"692ca66b-97c3-482e-a5dc-e9bcd77774e5\") " pod="openshift-must-gather-bcqz5/must-gather-rjtdz"
May 29 11:19:50.649651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.649555    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bcqz5/must-gather-rjtdz"
May 29 11:19:50.781498 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:50.781468    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bcqz5/must-gather-rjtdz"]
May 29 11:19:50.784090 ip-10-0-142-12 kubenswrapper[2547]: W0529 11:19:50.784061    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod692ca66b_97c3_482e_a5dc_e9bcd77774e5.slice/crio-421a35519fba34941ea89b0bca53c349597fa88ed81d2b31bc5e5858a11c984a WatchSource:0}: Error finding container 421a35519fba34941ea89b0bca53c349597fa88ed81d2b31bc5e5858a11c984a: Status 404 returned error can't find the container with id 421a35519fba34941ea89b0bca53c349597fa88ed81d2b31bc5e5858a11c984a
May 29 11:19:51.487569 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:51.487534    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bcqz5/must-gather-rjtdz" event={"ID":"692ca66b-97c3-482e-a5dc-e9bcd77774e5","Type":"ContainerStarted","Data":"421a35519fba34941ea89b0bca53c349597fa88ed81d2b31bc5e5858a11c984a"}
May 29 11:19:52.494225 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:52.494177    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bcqz5/must-gather-rjtdz" event={"ID":"692ca66b-97c3-482e-a5dc-e9bcd77774e5","Type":"ContainerStarted","Data":"b3e3f77afb412188ceba2ad936f6002850e119a659f2db153326bbc8fa9fa074"}
May 29 11:19:52.494779 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:52.494232    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bcqz5/must-gather-rjtdz" event={"ID":"692ca66b-97c3-482e-a5dc-e9bcd77774e5","Type":"ContainerStarted","Data":"afcfdc9fb2e3350524617aac0e7dcd9968b316c0a62ad4f78ce958e98a1bd0d2"}
May 29 11:19:53.387651 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:53.387616    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v8pkg_a07d3f20-9317-46cb-aab7-097ac29c23a3/global-pull-secret-syncer/0.log"
May 29 11:19:53.479467 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:53.479428    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nxddb_a8f095a4-c23f-42ed-b8c5-c76aea6f5bb6/konnectivity-agent/0.log"
May 29 11:19:53.591273 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:53.591219    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-12.ec2.internal_f693add312221a196297edaec6e814ca/haproxy/0.log"
May 29 11:19:55.168062 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.167976    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-bcqz5/must-gather-rjtdz" podStartSLOduration=4.202348199 podStartE2EDuration="5.16795323s" podCreationTimestamp="2026-05-29 11:19:50 +0000 UTC" firstStartedPulling="2026-05-29 11:19:50.786048228 +0000 UTC m=+1769.662610434" lastFinishedPulling="2026-05-29 11:19:51.751653255 +0000 UTC m=+1770.628215465" observedRunningTime="2026-05-29 11:19:52.513952046 +0000 UTC m=+1771.390514275" watchObservedRunningTime="2026-05-29 11:19:55.16795323 +0000 UTC m=+1774.044515456"
May 29 11:19:55.169382 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.169350    2547 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-56jlj/must-gather-dqmmn"]
May 29 11:19:55.169820 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.169773    2547 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-56jlj/must-gather-dqmmn" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" containerName="copy" containerID="cri-o://e02e9e5fdc464c99aa43ced7c30ae0e8726ec21121b37745562ef1c10919a557" gracePeriod=2
May 29 11:19:55.172028 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.172000    2547 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-56jlj/must-gather-dqmmn"]
May 29 11:19:55.172201 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.172113    2547 status_manager.go:895] "Failed to get status for pod" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" pod="openshift-must-gather-56jlj/must-gather-dqmmn" err="pods \"must-gather-dqmmn\" is forbidden: User \"system:node:ip-10-0-142-12.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-56jlj\": no relationship found between node 'ip-10-0-142-12.ec2.internal' and this object"
May 29 11:19:55.517786 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.517729    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-56jlj_must-gather-dqmmn_a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d/copy/0.log"
May 29 11:19:55.518281 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.518216    2547 generic.go:358] "Generic (PLEG): container finished" podID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" containerID="e02e9e5fdc464c99aa43ced7c30ae0e8726ec21121b37745562ef1c10919a557" exitCode=143
May 29 11:19:55.518428 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.518387    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="483f6d013f812a1cc84649bbf09340d25a00631f308ba5779f5c3b0d889ddebc"
May 29 11:19:55.533627 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.533525    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-56jlj_must-gather-dqmmn_a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d/copy/0.log"
May 29 11:19:55.534533 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.534216    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:55.537534 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.537480    2547 status_manager.go:895] "Failed to get status for pod" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" pod="openshift-must-gather-56jlj/must-gather-dqmmn" err="pods \"must-gather-dqmmn\" is forbidden: User \"system:node:ip-10-0-142-12.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-56jlj\": no relationship found between node 'ip-10-0-142-12.ec2.internal' and this object"
May 29 11:19:55.718424 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.717780    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-must-gather-output\") pod \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\" (UID: \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\") "
May 29 11:19:55.718781 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.718221    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" (UID: "a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 11:19:55.719053 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.719020    2547 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trwrm\" (UniqueName: \"kubernetes.io/projected/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-kube-api-access-trwrm\") pod \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\" (UID: \"a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d\") "
May 29 11:19:55.720164 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.720123    2547 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-must-gather-output\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:55.731407 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.731326    2547 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-kube-api-access-trwrm" (OuterVolumeSpecName: "kube-api-access-trwrm") pod "a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" (UID: "a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d"). InnerVolumeSpecName "kube-api-access-trwrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 11:19:55.735475 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.733393    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" path="/var/lib/kubelet/pods/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d/volumes"
May 29 11:19:55.821137 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:55.821073    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trwrm\" (UniqueName: \"kubernetes.io/projected/a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d-kube-api-access-trwrm\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\""
May 29 11:19:56.488348 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.488300    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6fe6de1b-b96c-486e-8511-ef57291c1233/alertmanager/0.log"
May 29 11:19:56.523210 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.523171    2547 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56jlj/must-gather-dqmmn"
May 29 11:19:56.531328 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.531296    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6fe6de1b-b96c-486e-8511-ef57291c1233/config-reloader/0.log"
May 29 11:19:56.571800 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.571771    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6fe6de1b-b96c-486e-8511-ef57291c1233/kube-rbac-proxy-web/0.log"
May 29 11:19:56.607985 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.607949    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6fe6de1b-b96c-486e-8511-ef57291c1233/kube-rbac-proxy/0.log"
May 29 11:19:56.647025 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.646996    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6fe6de1b-b96c-486e-8511-ef57291c1233/kube-rbac-proxy-metric/0.log"
May 29 11:19:56.676877 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.676852    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6fe6de1b-b96c-486e-8511-ef57291c1233/prom-label-proxy/0.log"
May 29 11:19:56.706133 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.706103    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6fe6de1b-b96c-486e-8511-ef57291c1233/init-config-reloader/0.log"
May 29 11:19:56.785626 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.785588    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-grsf9_60068c98-1803-4dc6-8bc6-b17d7a9fa16e/kube-state-metrics/0.log"
May 29 11:19:56.811195 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.811158    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-grsf9_60068c98-1803-4dc6-8bc6-b17d7a9fa16e/kube-rbac-proxy-main/0.log"
May 29 11:19:56.839532 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.839483    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-grsf9_60068c98-1803-4dc6-8bc6-b17d7a9fa16e/kube-rbac-proxy-self/0.log"
May 29 11:19:56.908772 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:56.908740    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-69ww8_5f0c431e-a55d-4dff-8673-5d6e870b4bed/monitoring-plugin/0.log"
May 29 11:19:57.119666 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.119582    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-llk2k_0859261f-38fc-47b2-ae8f-23f946e87d42/node-exporter/0.log"
May 29 11:19:57.143560 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.143528    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-llk2k_0859261f-38fc-47b2-ae8f-23f946e87d42/kube-rbac-proxy/0.log"
May 29 11:19:57.169288 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.169261    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-llk2k_0859261f-38fc-47b2-ae8f-23f946e87d42/init-textfile/0.log"
May 29 11:19:57.199599 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.199569    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-k2hcg_f340ea9b-0c5c-45d2-b0f6-db1dc01354cc/kube-rbac-proxy-main/0.log"
May 29 11:19:57.240677 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.240649    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-k2hcg_f340ea9b-0c5c-45d2-b0f6-db1dc01354cc/kube-rbac-proxy-self/0.log"
May 29 11:19:57.280796 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.280762    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-k2hcg_f340ea9b-0c5c-45d2-b0f6-db1dc01354cc/openshift-state-metrics/0.log"
May 29 11:19:57.502260 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.502160    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-mcl5k_82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb/prometheus-operator/0.log"
May 29 11:19:57.532189 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.532151    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-mcl5k_82d2b2e9-6ba4-4cfb-b7b5-4ccf0d94cbfb/kube-rbac-proxy/0.log"
May 29 11:19:57.563546 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.563506    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-9m5r8_1621fbfd-dd5c-42c9-b036-4410ffad4608/prometheus-operator-admission-webhook/0.log"
May 29 11:19:57.596480 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.596441    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-859cccc5c4-lc5sn_ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b/telemeter-client/0.log"
May 29 11:19:57.622847 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.622812    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-859cccc5c4-lc5sn_ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b/reload/0.log"
May 29 11:19:57.649561 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.649533    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-859cccc5c4-lc5sn_ea4cd5e5-96e0-4c9d-8a0f-77f4e39a1b8b/kube-rbac-proxy/0.log"
May 29 11:19:57.702047 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.702009    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8f647b-stcr9_258ae73c-dce5-4991-8562-cdf47bd22f50/thanos-query/0.log"
May 29 11:19:57.732673 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.732634    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8f647b-stcr9_258ae73c-dce5-4991-8562-cdf47bd22f50/kube-rbac-proxy-web/0.log"
May 29 11:19:57.764503 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.764468    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8f647b-stcr9_258ae73c-dce5-4991-8562-cdf47bd22f50/kube-rbac-proxy/0.log"
May 29 11:19:57.793171 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.793136    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8f647b-stcr9_258ae73c-dce5-4991-8562-cdf47bd22f50/prom-label-proxy/0.log"
May 29 11:19:57.822205 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.822170    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8f647b-stcr9_258ae73c-dce5-4991-8562-cdf47bd22f50/kube-rbac-proxy-rules/0.log"
May 29 11:19:57.856550 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:57.856516    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8f647b-stcr9_258ae73c-dce5-4991-8562-cdf47bd22f50/kube-rbac-proxy-metrics/0.log"
May 29 11:19:59.876621 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:19:59.876573    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-c7hjh_6b150d8b-02db-466e-bd9f-8d875e33a0e2/download-server/0.log"
May 29 11:20:00.036092 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.036060    2547 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"]
May 29 11:20:00.036432 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.036407    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" containerName="gather"
May 29 11:20:00.036432 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.036422    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" containerName="gather"
May 29 11:20:00.036432 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.036431    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" containerName="copy"
May 29 11:20:00.036572 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.036437    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" containerName="copy"
May 29 11:20:00.036572 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.036492    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" containerName="copy"
May 29 11:20:00.036572 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.036504    2547 memory_manager.go:356] "RemoveStaleState removing state" podUID="a05e3bd1-8c3e-45ef-9f26-e4b3b1d8f94d" containerName="gather"
May 29 11:20:00.042340 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.042293    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.051083 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.051048    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"]
May 29 11:20:00.060017 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.059988    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-sys\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.060193 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.060032    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2t4\" (UniqueName: \"kubernetes.io/projected/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-kube-api-access-xk2t4\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.060193 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.060054    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-lib-modules\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.060193 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.060161    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-proc\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.060343 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.060203    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-podres\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.160764 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.160673    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-sys\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.160764 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.160725    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2t4\" (UniqueName: \"kubernetes.io/projected/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-kube-api-access-xk2t4\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.160961 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.160800    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-sys\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.160961 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.160844    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-lib-modules\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.160961 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.160883    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-proc\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.160961 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.160901    2547 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-podres\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.161108 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.160979    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-proc\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.161108 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.161014    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-lib-modules\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.161108 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.161048    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-podres\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.171454 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.171424    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2t4\" (UniqueName: \"kubernetes.io/projected/e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90-kube-api-access-xk2t4\") pod \"perf-node-gather-daemonset-8pfrn\" (UID: \"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90\") " pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.353455 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.353413    2547 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:00.509949 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.509922    2547 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"]
May 29 11:20:00.540481 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:00.540450    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn" event={"ID":"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90","Type":"ContainerStarted","Data":"71bde76f58b66c9521091be3d15e1229c9006beed32b3a5809aab36eec93a9cd"}
May 29 11:20:01.193598 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:01.193562    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wzwmb_8da26b43-5619-444b-929c-a54e16f29818/dns/0.log"
May 29 11:20:01.227633 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:01.227589    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wzwmb_8da26b43-5619-444b-929c-a54e16f29818/kube-rbac-proxy/0.log"
May 29 11:20:01.289586 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:01.289549    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ghsp2_ad2f5640-3ce2-4e84-acbb-f3f8b6ed0041/dns-node-resolver/0.log"
May 29 11:20:01.545654 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:01.545610    2547 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn" event={"ID":"e2c6ff65-0b50-4ea5-aca7-ab768ea9ac90","Type":"ContainerStarted","Data":"c883b5b76ff352fbb3ef0cab0805cee91727b6cda7e6aa37add773112e1e7c23"}
May 29 11:20:01.546024 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:01.546000    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:01.570409 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:01.570356    2547 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn" podStartSLOduration=1.570340924 podStartE2EDuration="1.570340924s" podCreationTimestamp="2026-05-29 11:20:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 11:20:01.569646115 +0000 UTC m=+1780.446208346" watchObservedRunningTime="2026-05-29 11:20:01.570340924 +0000 UTC m=+1780.446903152"
May 29 11:20:01.839149 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:01.839069    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2n89j_e8e00f00-647c-463b-b317-aa670bde2fd4/node-ca/0.log"
May 29 11:20:03.170830 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:03.170794    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5dlbc_5c4e7979-7e81-4939-a895-b934805b1275/serve-healthcheck-canary/0.log"
May 29 11:20:03.679033 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:03.678998    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wlpw2_afddd537-f5d6-401f-95d7-5ad4c38ff18b/kube-rbac-proxy/0.log"
May 29 11:20:03.703602 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:03.703569    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wlpw2_afddd537-f5d6-401f-95d7-5ad4c38ff18b/exporter/0.log"
May 29 11:20:03.737845 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:03.737807    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wlpw2_afddd537-f5d6-401f-95d7-5ad4c38ff18b/extractor/0.log"
May 29 11:20:07.560962 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:07.560933    2547 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bcqz5/perf-node-gather-daemonset-8pfrn"
May 29 11:20:09.269878 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:09.269842    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-97zrn_4dcaa3dd-8715-4b80-84e1-5000e413b7e5/kube-multus/0.log"
May 29 11:20:09.751019 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:09.750994    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qm79t_c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7/kube-multus-additional-cni-plugins/0.log"
May 29 11:20:09.784450 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:09.784420    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qm79t_c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7/egress-router-binary-copy/0.log"
May 29 11:20:09.820098 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:09.820064    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qm79t_c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7/cni-plugins/0.log"
May 29 11:20:09.844950 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:09.844861    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qm79t_c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7/bond-cni-plugin/0.log"
May 29 11:20:09.874747 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:09.874720    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qm79t_c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7/routeoverride-cni/0.log"
May 29 11:20:09.899111 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:09.899074    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qm79t_c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7/whereabouts-cni-bincopy/0.log"
May 29 11:20:09.922397 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:09.922367    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qm79t_c14ee7fa-f840-4bc7-b2a5-28e8d5037fe7/whereabouts-cni/0.log"
May 29 11:20:10.030402 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:10.030360    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fqncf_7883155e-3d26-41dc-8c39-a954501459b5/network-metrics-daemon/0.log"
May 29 11:20:10.051351 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:10.051323    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fqncf_7883155e-3d26-41dc-8c39-a954501459b5/kube-rbac-proxy/0.log"
May 29 11:20:11.424077 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:11.424048    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-controller/0.log"
May 29 11:20:11.441489 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:11.441457    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/0.log"
May 29 11:20:11.458964 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:11.458922    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovn-acl-logging/1.log"
May 29 11:20:11.482574 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:11.482540    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/kube-rbac-proxy-node/0.log"
May 29 11:20:11.512677 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:11.512644    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/kube-rbac-proxy-ovn-metrics/0.log"
May 29 11:20:11.548157 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:11.548118    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/northd/0.log"
May 29 11:20:11.573073 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:11.573049    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/nbdb/0.log"
May 29 11:20:11.596899 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:11.596852    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/sbdb/0.log"
May 29 11:20:11.794780 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:11.794735    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c8682_a152d23c-60e9-432e-98c3-799273b117fd/ovnkube-controller/0.log"
May 29 11:20:12.831851 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:12.831817    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4524p_532d0df8-fdbb-40c5-8f95-d6f2975fc47e/network-check-target-container/0.log"
May 29 11:20:13.767491 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:13.767461    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zlmdr_d00bb862-d044-4f60-8f8e-75f3e539d7a9/iptables-alerter/0.log"
May 29 11:20:14.385196 ip-10-0-142-12 kubenswrapper[2547]: I0529 11:20:14.385161    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-x84w9_d18acbe7-c88b-44c1-9f3e-a0c32828dbb8/tuned/0.log"