May 26 17:04:19.157365 ip-10-0-129-109 systemd[1]: Starting Kubernetes Kubelet...
May 26 17:04:19.587837 ip-10-0-129-109 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 17:04:19.587837 ip-10-0-129-109 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 17:04:19.587837 ip-10-0-129-109 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 17:04:19.587837 ip-10-0-129-109 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 17:04:19.587837 ip-10-0-129-109 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 17:04:19.589462 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.589370    2571 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 17:04:19.592347 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592332    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 17:04:19.592347 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592347    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592351    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592354    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592356    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592359    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592362    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592364    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592368    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592371    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592374    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592377    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592380    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592382    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592385    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592387    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592390    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592393    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592395    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592398    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592400    2571 feature_gate.go:328] unrecognized feature gate: Example
May 26 17:04:19.592409 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592403    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592405    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592408    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592411    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592414    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592416    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592419    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592422    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592425    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592427    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592430    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592432    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592435    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592437    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592439    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592442    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592445    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592447    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592450    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592452    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 17:04:19.592879 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592455    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592457    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592460    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592463    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592465    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592468    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592471    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592474    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592476    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592479    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592481    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592485    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592489    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592492    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592496    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592499    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592504    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592507    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592511    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 17:04:19.593400 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592520    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592523    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592526    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592529    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592533    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592536    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592539    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592542    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592544    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592547    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592550    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592553    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592555    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592558    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592560    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592563    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592566    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592569    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592571    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592574    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 17:04:19.593885 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592576    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592579    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592581    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592584    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592588    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592591    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.592997    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593003    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593006    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593028    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593032    2571 feature_gate.go:328] unrecognized feature gate: Example
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593034    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593037    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593040    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593042    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593045    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593048    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593051    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593053    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593056    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 17:04:19.594373 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593058    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593061    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593064    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593066    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593069    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593072    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593074    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593077    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593085    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593088    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593091    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593093    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593096    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593098    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593101    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593104    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593107    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593109    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593112    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593115    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 17:04:19.594847 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593118    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593120    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593123    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593126    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593128    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593131    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593133    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593136    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593138    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593140    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593143    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593145    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593148    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593150    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593153    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593155    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593159    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593161    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593164    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 17:04:19.595378 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593169    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593172    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593181    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593184    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593187    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593189    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593192    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593194    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593196    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593199    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593201    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593204    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593206    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593209    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593211    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593215    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593218    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593222    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593224    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593227    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 17:04:19.595882 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593229    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593231    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593234    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593236    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593239    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593241    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593244    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593247    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593250    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593252    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593255    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593260    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593263    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593344    2571 flags.go:64] FLAG: --address="0.0.0.0"
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593352    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593362    2571 flags.go:64] FLAG: --anonymous-auth="true"
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593366    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593371    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593374    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593378    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593382    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 17:04:19.596382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593386    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593389    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593392    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593396    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593399    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593403    2571 flags.go:64] FLAG: --cgroup-root=""
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593406    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593409    2571 flags.go:64] FLAG: --client-ca-file=""
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593412    2571 flags.go:64] FLAG: --cloud-config=""
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593415    2571 flags.go:64] FLAG: --cloud-provider="external"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593418    2571 flags.go:64] FLAG: --cluster-dns="[]"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593426    2571 flags.go:64] FLAG: --cluster-domain=""
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593429    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593432    2571 flags.go:64] FLAG: --config-dir=""
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593435    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593439    2571 flags.go:64] FLAG: --container-log-max-files="5"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593443    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593446    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593449    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593452    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593455    2571 flags.go:64] FLAG: --contention-profiling="false"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593458    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593464    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593467    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593470    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 17:04:19.596889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593474    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593477    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593480    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593483    2571 flags.go:64] FLAG: --enable-load-reader="false"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593486    2571 flags.go:64] FLAG: --enable-server="true"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593489    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593494    2571 flags.go:64] FLAG: --event-burst="100"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593498    2571 flags.go:64] FLAG: --event-qps="50"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593501    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593504    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593507    2571 flags.go:64] FLAG: --eviction-hard=""
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593510    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593513    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593516    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593519    2571 flags.go:64] FLAG: --eviction-soft=""
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593522    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593525    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593528    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593530    2571 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593533    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593536    2571 flags.go:64] FLAG: --fail-swap-on="true"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593539    2571 flags.go:64] FLAG: --feature-gates=""
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593543    2571 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593546    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593549    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 17:04:19.597512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593552    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593555    2571 flags.go:64] FLAG: --healthz-port="10248"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593558    2571 flags.go:64] FLAG: --help="false"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593561    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-129-109.ec2.internal"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593567    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593570    2571 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593573    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593576    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593580    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593583    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593587    2571 flags.go:64] FLAG: --image-service-endpoint=""
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593589    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593592    2571 flags.go:64] FLAG: --kube-api-burst="100"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593595    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593598    2571 flags.go:64] FLAG: --kube-api-qps="50"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593601    2571 flags.go:64] FLAG: --kube-reserved=""
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593604    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593607    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593610    2571 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593613    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593615    2571 flags.go:64] FLAG: --lock-file=""
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593618    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593621    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593624    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 17:04:19.598158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593630    2571 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593633    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593636    2571 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593639    2571 flags.go:64] FLAG: --logging-format="text"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593642    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593645    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593648    2571 flags.go:64] FLAG: --manifest-url=""
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593651    2571 flags.go:64] FLAG: --manifest-url-header=""
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593655    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593658    2571 flags.go:64] FLAG: --max-open-files="1000000"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593662    2571 flags.go:64] FLAG: --max-pods="110"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593664    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593669    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593671    2571 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593675    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593678    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593681    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593684    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593691    2571 flags.go:64] FLAG: --node-status-max-images="50"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593698    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593701    2571 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593704    2571 flags.go:64] FLAG: --pod-cidr=""
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593707    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 17:04:19.598731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593712    2571 flags.go:64] FLAG: --pod-manifest-path=""
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593715    2571 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593719    2571 flags.go:64] FLAG: --pods-per-core="0"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593721    2571 flags.go:64] FLAG: --port="10250"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593724    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593727    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01a33edfd02e1ea37"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593730    2571 flags.go:64] FLAG: --qos-reserved=""
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593733    2571 flags.go:64] FLAG: --read-only-port="10255"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593736    2571 flags.go:64] FLAG: --register-node="true"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593739    2571 flags.go:64] FLAG: --register-schedulable="true"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593742    2571 flags.go:64] FLAG: --register-with-taints=""
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593746    2571 flags.go:64] FLAG: --registry-burst="10"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593749    2571 flags.go:64] FLAG: --registry-qps="5"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593751    2571 flags.go:64] FLAG: --reserved-cpus=""
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593754    2571 flags.go:64] FLAG: --reserved-memory=""
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593760    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593763    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593767    2571 flags.go:64] FLAG: --rotate-certificates="false"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593770    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593772    2571 flags.go:64] FLAG: --runonce="false"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593775    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593778    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593782    2571 flags.go:64] FLAG: --seccomp-default="false"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593785    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593788    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593791    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 17:04:19.599307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593794    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593798    2571 flags.go:64] FLAG: --storage-driver-password="root"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593801    2571 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593804    2571 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593807    2571 flags.go:64] FLAG: --storage-driver-user="root"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593809    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593812    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593815    2571 flags.go:64] FLAG: --system-cgroups=""
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593818    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593823    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593826    2571 flags.go:64] FLAG: --tls-cert-file=""
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593829    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593833    2571 flags.go:64] FLAG: --tls-min-version=""
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593835    2571 flags.go:64] FLAG: --tls-private-key-file=""
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593838    2571 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593841    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593844    2571 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593847    2571 flags.go:64] FLAG: --v="2"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593851    2571 flags.go:64] FLAG: --version="false"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593856    2571 flags.go:64] FLAG: --vmodule=""
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593860    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.593863    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593964    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593968    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 17:04:19.599931 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593971    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593974    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593977    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593980    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593984    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593987    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593990    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593993    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593996    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.593999    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594001    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594004    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594007    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594027    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594031    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594034    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594037    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594040    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594043    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 17:04:19.600694 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594045    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594048    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594051    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594053    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594056    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594059    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594062    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594065    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594067    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594070    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594073    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594075    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594078    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594080    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594083    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594085    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594088    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594092    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594095    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594097    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 17:04:19.601456 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594100    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594103    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594105    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594109    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594113    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594115    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594119    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594122    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594125    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594127    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594130    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594132    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594135    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594137    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594140    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594142    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594145    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594149    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594152    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 17:04:19.601956 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594154    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594157    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594160    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594162    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594165    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594167    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594170    2571 feature_gate.go:328] unrecognized feature gate: Example
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594172    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594175    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594177    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594181    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594184    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594186    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594189    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594192    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594195    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594198    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594200    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594203    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594205    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 17:04:19.602467 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594209    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 17:04:19.602969 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594211    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 17:04:19.602969 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594214    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 17:04:19.602969 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594216    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 17:04:19.602969 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594219    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 17:04:19.602969 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.594221    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 17:04:19.602969 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.594968    2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 17:04:19.604883 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.604862    2571 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 17:04:19.604920 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.604884    2571 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 17:04:19.604947 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604934    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 17:04:19.604947 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604939    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 17:04:19.604947 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604944    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 17:04:19.604947 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604947    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604950    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604953    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604957    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604960    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604963    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604966    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604969    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604971    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604974    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604977    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604980    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604982    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604985    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604989    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604994    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.604998    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605002    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605005    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 17:04:19.605055 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605021    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605025    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605027    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605030    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605033    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605036    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605039    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605041    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605044    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605047    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605049    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605052    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605055    2571 feature_gate.go:328] unrecognized feature gate: Example
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605058    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605060    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605063    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605066    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605069    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605072    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605075    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 17:04:19.605527 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605077    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605081    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605084    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605086    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605089    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605091    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605094    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605096    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605099    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605101    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605104    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605106    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605109    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605111    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605114    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605117    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605120    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605122    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605125    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605128    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 17:04:19.606004 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605130    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605133    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605135    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605138    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605140    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605143    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605146    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605148    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605151    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605153    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605156    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605158    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605161    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605163    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605167    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605170    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605172    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605175    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605177    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605180    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 17:04:19.606507 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605182    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605185    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605187    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605190    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.605195    2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605297    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605303    2571 feature_gate.go:328] unrecognized feature gate: Example
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605307    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605310    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605312    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605315    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605318    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605321    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605323    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605326    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 17:04:19.607072 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605330    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605334    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605337    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605340    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605343    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605345    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605348    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605350    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605353    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605355    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605358    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605360    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605363    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605366    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605369    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605371    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605373    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605376    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605378    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605381    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 17:04:19.607450 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605383    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605386    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605389    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605392    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605395    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605398    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605400    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605403    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605405    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605408    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605411    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605413    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605416    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605418    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605421    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605423    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605426    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605428    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605431    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605433    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 17:04:19.607940 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605436    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605440    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605443    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605445    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605448    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605451    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605454    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605456    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605459    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605461    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605464    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605466    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605469    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605472    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605475    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605478    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605481    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605483    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605486    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 17:04:19.608445 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605489    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605492    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605494    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605497    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605500    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605502    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605505    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605507    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605510    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605512    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605515    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605518    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605520    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605522    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605525    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605528    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 17:04:19.608901 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:19.605531    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 17:04:19.609325 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.605536    2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 17:04:19.609325 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.606271    2571 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 17:04:19.609325 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.608566    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 17:04:19.609523 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.609510    2571 server.go:1019] "Starting client certificate rotation"
May 26 17:04:19.609620 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.609606    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 17:04:19.609649 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.609644    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 17:04:19.634008 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.633986    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 17:04:19.635893 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.635874    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 17:04:19.651493 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.651471    2571 log.go:25] "Validated CRI v1 runtime API"
May 26 17:04:19.656369 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.656352    2571 log.go:25] "Validated CRI v1 image API"
May 26 17:04:19.657720 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.657703    2571 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 17:04:19.664353 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.664332    2571 fs.go:135] Filesystem UUIDs: map[0ed6c077-e0fd-43b1-af30-b930f68521a2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a67fad62-bf0e-472d-aa4d-65847953318d:/dev/nvme0n1p4]
May 26 17:04:19.664419 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.664353    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 17:04:19.665255 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.665237    2571 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 17:04:19.668591 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.668488    2571 manager.go:217] Machine: {Timestamp:2026-05-26 17:04:19.667985454 +0000 UTC m=+0.393074946 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098385 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b00ff9cdb4eac986ffb793c1cba35 SystemUUID:ec2b00ff-9cdb-4eac-986f-fb793c1cba35 BootID:f1a24534-6d06-409c-ba4b-b1f3aa3ab2a9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8d:06:b4:4d:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8d:06:b4:4d:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:94:bf:04:02:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 17:04:19.669471 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.669460    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 17:04:19.669591 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.669579    2571 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 17:04:19.672785 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.672764    2571 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 17:04:19.672925 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.672793    2571 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-109.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 17:04:19.672974 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.672934    2571 topology_manager.go:138] "Creating topology manager with none policy"
May 26 17:04:19.672974 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.672944    2571 container_manager_linux.go:306] "Creating device plugin manager"
May 26 17:04:19.672974 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.672951    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 17:04:19.673766 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.673755    2571 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 17:04:19.675524 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.675512    2571 state_mem.go:36] "Initialized new in-memory state store"
May 26 17:04:19.675643 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.675634    2571 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 17:04:19.677986 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.677976    2571 kubelet.go:491] "Attempting to sync node with API server"
May 26 17:04:19.678030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.677995    2571 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 17:04:19.678030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.678020    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 17:04:19.678030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.678029    2571 kubelet.go:397] "Adding apiserver pod source"
May 26 17:04:19.678124 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.678037    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 17:04:19.679131 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.679120    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 17:04:19.679179 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.679139    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 17:04:19.682645 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.682626    2571 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 17:04:19.684562 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.684546    2571 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 17:04:19.686491 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686477    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 17:04:19.686535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686494    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 17:04:19.686535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686501    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 17:04:19.686535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686507    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 17:04:19.686535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686516    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 17:04:19.686535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686523    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 17:04:19.686535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686529    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 17:04:19.686535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686534    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 17:04:19.686752 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686541    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 17:04:19.686752 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686547    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 17:04:19.686752 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686557    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 17:04:19.686752 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.686563    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 17:04:19.688312 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.688299    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 17:04:19.688355 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.688315    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 17:04:19.689628 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.689605    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 17:04:19.689679 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.689636    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 17:04:19.691557 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.691544    2571 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 17:04:19.691638 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.691586    2571 server.go:1295] "Started kubelet"
May 26 17:04:19.691721 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.691649    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 17:04:19.691801 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.691682    2571 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 17:04:19.691835 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.691822    2571 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 17:04:19.692439 ip-10-0-129-109 systemd[1]: Started Kubernetes Kubelet.
May 26 17:04:19.693059 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.693046    2571 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 17:04:19.693797 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.693785    2571 server.go:317] "Adding debug handlers to kubelet server"
May 26 17:04:19.699915 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.699893    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 17:04:19.700131 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.700109    2571 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-109.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 17:04:19.700604 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.700583    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 17:04:19.700893 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.700874    2571 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 17:04:19.701023 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.700997    2571 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 17:04:19.701244 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.701229    2571 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 17:04:19.701378 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.701365    2571 reconstruct.go:97] "Volume reconstruction finished"
May 26 17:04:19.701378 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.701379    2571 reconciler.go:26] "Reconciler: start to sync state"
May 26 17:04:19.701710 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.701624    2571 factory.go:55] Registering systemd factory
May 26 17:04:19.701710 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.701644    2571 factory.go:223] Registration of the systemd container factory successfully
May 26 17:04:19.701845 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.700175    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-109.ec2.internal.18b32c5deece541e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-109.ec2.internal,UID:ip-10-0-129-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-109.ec2.internal,},FirstTimestamp:2026-05-26 17:04:19.691557918 +0000 UTC m=+0.416647410,LastTimestamp:2026-05-26 17:04:19.691557918 +0000 UTC m=+0.416647410,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-109.ec2.internal,}"
May 26 17:04:19.701845 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.701777    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:19.701954 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.701903    2571 factory.go:153] Registering CRI-O factory
May 26 17:04:19.701954 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.701917    2571 factory.go:223] Registration of the crio container factory successfully
May 26 17:04:19.702062 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.701968    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 17:04:19.702062 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.701988    2571 factory.go:103] Registering Raw factory
May 26 17:04:19.702062 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.702004    2571 manager.go:1196] Started watching for new ooms in manager
May 26 17:04:19.702422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.702406    2571 manager.go:319] Starting recovery of all containers
May 26 17:04:19.703181 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.703157    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5r8k"
May 26 17:04:19.703809 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.703714    2571 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 17:04:19.706764 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.706736    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 26 17:04:19.706871 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.706856    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 26 17:04:19.711445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.711421    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5r8k"
May 26 17:04:19.713194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.713178    2571 manager.go:324] Recovery completed
May 26 17:04:19.714878 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.714857    2571 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
May 26 17:04:19.717751 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.717739    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 17:04:19.719378 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.719362    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientMemory"
May 26 17:04:19.719436 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.719392    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasNoDiskPressure"
May 26 17:04:19.719436 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.719407    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientPID"
May 26 17:04:19.719900 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.719886    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 17:04:19.719900 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.719898    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 17:04:19.720007 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.719916    2571 state_mem.go:36] "Initialized new in-memory state store"
May 26 17:04:19.721354 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.721281    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-109.ec2.internal.18b32c5df076deb5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-109.ec2.internal,UID:ip-10-0-129-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-109.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-109.ec2.internal,},FirstTimestamp:2026-05-26 17:04:19.719380661 +0000 UTC m=+0.444470154,LastTimestamp:2026-05-26 17:04:19.719380661 +0000 UTC m=+0.444470154,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-109.ec2.internal,}"
May 26 17:04:19.722360 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.722345    2571 policy_none.go:49] "None policy: Start"
May 26 17:04:19.722360 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.722362    2571 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 17:04:19.722441 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.722373    2571 state_mem.go:35] "Initializing new in-memory state store"
May 26 17:04:19.769585 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.769567    2571 manager.go:341] "Starting Device Plugin manager"
May 26 17:04:19.777987 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.769632    2571 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 17:04:19.777987 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.769643    2571 server.go:85] "Starting device plugin registration server"
May 26 17:04:19.777987 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.769857    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 17:04:19.777987 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.769867    2571 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 17:04:19.777987 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.769967    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 17:04:19.777987 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.770073    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 17:04:19.777987 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.770082    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 17:04:19.777987 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.770861    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 17:04:19.777987 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.770905    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:19.828376 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.828340    2571 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 17:04:19.829715 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.829694    2571 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 17:04:19.829715 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.829721    2571 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 17:04:19.829868 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.829743    2571 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 17:04:19.829868 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.829753    2571 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 17:04:19.829868 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.829855    2571 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 17:04:19.832456 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.832437    2571 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 17:04:19.870510 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.870458    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 17:04:19.871408 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.871393    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientMemory"
May 26 17:04:19.871477 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.871423    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasNoDiskPressure"
May 26 17:04:19.871477 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.871435    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientPID"
May 26 17:04:19.871477 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.871458    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-109.ec2.internal"
May 26 17:04:19.878351 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.878337    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-109.ec2.internal"
May 26 17:04:19.878405 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.878359    2571 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-109.ec2.internal\": node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:19.905331 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.905308    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:19.930343 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.930321    2571 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal"]
May 26 17:04:19.930405 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.930385    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 17:04:19.932409 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.932393    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientMemory"
May 26 17:04:19.932516 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.932420    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasNoDiskPressure"
May 26 17:04:19.932516 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.932432    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientPID"
May 26 17:04:19.933878 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.933864    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 17:04:19.934056 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.934041    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal"
May 26 17:04:19.934105 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.934073    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 17:04:19.934619 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.934603    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientMemory"
May 26 17:04:19.934698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.934634    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasNoDiskPressure"
May 26 17:04:19.934698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.934645    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientPID"
May 26 17:04:19.934698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.934611    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientMemory"
May 26 17:04:19.934846 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.934703    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasNoDiskPressure"
May 26 17:04:19.934846 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.934718    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientPID"
May 26 17:04:19.936063 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.936047    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal"
May 26 17:04:19.936155 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.936072    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 17:04:19.936713 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.936698    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientMemory"
May 26 17:04:19.936775 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.936726    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasNoDiskPressure"
May 26 17:04:19.936775 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:19.936742    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeHasSufficientPID"
May 26 17:04:19.961643 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.961626    2571 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-109.ec2.internal\" not found" node="ip-10-0-129-109.ec2.internal"
May 26 17:04:19.965023 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:19.964984    2571 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-109.ec2.internal\" not found" node="ip-10-0-129-109.ec2.internal"
May 26 17:04:20.002765 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.002730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f09d26b0580610a005e903ab2681d50d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-109.ec2.internal\" (UID: \"f09d26b0580610a005e903ab2681d50d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.002765 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.002763    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a9722eed112a49f65691e180ade29ba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal\" (UID: \"3a9722eed112a49f65691e180ade29ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.002934 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.002778    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a9722eed112a49f65691e180ade29ba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal\" (UID: \"3a9722eed112a49f65691e180ade29ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.005824 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.005807    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.103709 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.103672    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f09d26b0580610a005e903ab2681d50d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-109.ec2.internal\" (UID: \"f09d26b0580610a005e903ab2681d50d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.103709 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.103709    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a9722eed112a49f65691e180ade29ba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal\" (UID: \"3a9722eed112a49f65691e180ade29ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.103893 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.103731    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a9722eed112a49f65691e180ade29ba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal\" (UID: \"3a9722eed112a49f65691e180ade29ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.103893 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.103787    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a9722eed112a49f65691e180ade29ba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal\" (UID: \"3a9722eed112a49f65691e180ade29ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.103893 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.103790    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a9722eed112a49f65691e180ade29ba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal\" (UID: \"3a9722eed112a49f65691e180ade29ba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.103893 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.103787    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f09d26b0580610a005e903ab2681d50d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-109.ec2.internal\" (UID: \"f09d26b0580610a005e903ab2681d50d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.106756 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.106735    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.207625 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.207550    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.264764 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.264730    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.267405 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.267379    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal"
May 26 17:04:20.308555 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.308526    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.409024 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.408984    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.509573 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.509532    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.609086 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.609056    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 17:04:20.609809 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.609221    2571 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 17:04:20.610161 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.610130    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.700836 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.700804    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 17:04:20.710480 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.710445    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.715425 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.715397    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 16:59:19 +0000 UTC" deadline="2027-11-14 02:36:24.900150758 +0000 UTC"
May 26 17:04:20.715425 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.715424    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12873h32m4.184731403s"
May 26 17:04:20.718943 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.718925    2571 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 17:04:20.739914 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.739892    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rclb9"
May 26 17:04:20.747564 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.747543    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rclb9"
May 26 17:04:20.794094 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.794029    2571 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 17:04:20.810550 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.810520    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.832366 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:20.832195    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09d26b0580610a005e903ab2681d50d.slice/crio-76df480db343cbe4a113ca87c069241229e2eeafc1cae3002fcb4be836697f9e WatchSource:0}: Error finding container 76df480db343cbe4a113ca87c069241229e2eeafc1cae3002fcb4be836697f9e: Status 404 returned error can't find the container with id 76df480db343cbe4a113ca87c069241229e2eeafc1cae3002fcb4be836697f9e
May 26 17:04:20.832599 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:20.832575    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9722eed112a49f65691e180ade29ba.slice/crio-9a8ea5394ddc47026b6d17fa5d96895171b7ff2fbfb1d9c3f1ebd1a3be87f6f3 WatchSource:0}: Error finding container 9a8ea5394ddc47026b6d17fa5d96895171b7ff2fbfb1d9c3f1ebd1a3be87f6f3: Status 404 returned error can't find the container with id 9a8ea5394ddc47026b6d17fa5d96895171b7ff2fbfb1d9c3f1ebd1a3be87f6f3
May 26 17:04:20.835766 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.835749    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 17:04:20.910858 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:20.910819    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-109.ec2.internal\" not found"
May 26 17:04:20.937842 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:20.937822    2571 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 17:04:21.001120 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.001091    2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal"
May 26 17:04:21.010237 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.010211    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 17:04:21.011961 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.011936    2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal"
May 26 17:04:21.023091 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.023071    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 17:04:21.224800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.224718    2571 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 17:04:21.679400 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.679353    2571 apiserver.go:52] "Watching apiserver"
May 26 17:04:21.690529 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.690497    2571 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 17:04:21.695086 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.695049    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-bz4lz","openshift-dns/node-resolver-m97tn","openshift-multus/multus-xrkfv","openshift-multus/network-metrics-daemon-lztjw","openshift-network-diagnostics/network-check-target-fjgrn","openshift-ovn-kubernetes/ovnkube-node-r56x2","kube-system/konnectivity-agent-r6jrs","kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h","openshift-image-registry/node-ca-6w2fz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal","openshift-multus/multus-additional-cni-plugins-8qh7j","openshift-network-operator/iptables-alerter-b498j"]
May 26 17:04:21.696994 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.696934    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:21.698318 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.698290    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:21.698454 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:21.698423    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:21.699831 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.699734    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.699831 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.699766    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 17:04:21.699990 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.699838    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 17:04:21.700113 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.700087    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-76xgl\""
May 26 17:04:21.702437 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.702255    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 17:04:21.703157 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.702638    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.703157 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.702720    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 17:04:21.703157 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.702938    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pfh9\""
May 26 17:04:21.703636 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.703612    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 17:04:21.703881 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.703797    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 17:04:21.703973 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.703883    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 17:04:21.703973 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.703889    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 17:04:21.703973 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.703942    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.705161 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.705141    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 17:04:21.705161 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.705153    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 17:04:21.705325 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.705283    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ghsbj\""
May 26 17:04:21.705325 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.705306    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 17:04:21.705563 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.705498    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.707336 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.707314    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 17:04:21.707435 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.707413    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.707546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.707528    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 17:04:21.707816 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.707798    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-szlr6\""
May 26 17:04:21.709132 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.708210    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 17:04:21.709132 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.708436    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h7hqc\""
May 26 17:04:21.709132 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.708594    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 17:04:21.709132 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.708797    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 17:04:21.709132 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.708986    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 17:04:21.709132 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.709119    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:21.709444 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.709138    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.709444 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:21.709190    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:21.710035 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.710000    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 17:04:21.710289 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.710271    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z64g9\""
May 26 17:04:21.710511 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.710493    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 17:04:21.711674 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.711502    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 17:04:21.711674 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.711558    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pplcb\""
May 26 17:04:21.711674 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.711586    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 17:04:21.711883 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.711868    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 17:04:21.712051 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712005    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712143    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-run-systemd\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712037    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712176    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-run-ovn\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712202    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-cni-netd\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712225    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-registration-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712249    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-sys-fs\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712272    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvt2p\" (UniqueName: \"kubernetes.io/projected/a13dbe45-e72d-4315-a2a1-c4736202fd9b-kube-api-access-kvt2p\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712296    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-slash\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712320    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-var-lib-openvswitch\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-run-openvswitch\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712402 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712391    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-run-ovn-kubernetes\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712413    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712436    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2753c993-0102-4e57-b557-f03934174f89-ovn-node-metrics-cert\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712458    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-device-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712488    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87203750-2eb1-4b8c-9c94-b7ca88363b40-hosts-file\") pod \"node-resolver-m97tn\" (UID: \"87203750-2eb1-4b8c-9c94-b7ca88363b40\") " pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712511    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-kubelet\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712546    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-etc-openvswitch\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712582    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712646    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2753c993-0102-4e57-b557-f03934174f89-env-overrides\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712697    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712727    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-systemd-units\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712753    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-run-netns\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712778    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712805    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2753c993-0102-4e57-b557-f03934174f89-ovnkube-script-lib\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712856    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87203750-2eb1-4b8c-9c94-b7ca88363b40-tmp-dir\") pod \"node-resolver-m97tn\" (UID: \"87203750-2eb1-4b8c-9c94-b7ca88363b40\") " pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712889    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8ce63184-976c-4e94-9b6c-c94a79abbf53-agent-certs\") pod \"konnectivity-agent-r6jrs\" (UID: \"8ce63184-976c-4e94-9b6c-c94a79abbf53\") " pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:21.712911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712910    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-node-log\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.713800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712955    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9m8\" (UniqueName: \"kubernetes.io/projected/2753c993-0102-4e57-b557-f03934174f89-kube-api-access-5d9m8\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.713800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.712992    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8ce63184-976c-4e94-9b6c-c94a79abbf53-konnectivity-ca\") pod \"konnectivity-agent-r6jrs\" (UID: \"8ce63184-976c-4e94-9b6c-c94a79abbf53\") " pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:21.713800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.713030    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-log-socket\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.713800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.713079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-cni-bin\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.713800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.713103    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2753c993-0102-4e57-b557-f03934174f89-ovnkube-config\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.713800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.713124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-socket-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.713800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.713160    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdh6m\" (UniqueName: \"kubernetes.io/projected/2067d39e-24b4-4da4-9631-2f927333a9ef-kube-api-access-hdh6m\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.713800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.713190    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6d7\" (UniqueName: \"kubernetes.io/projected/87203750-2eb1-4b8c-9c94-b7ca88363b40-kube-api-access-fr6d7\") pod \"node-resolver-m97tn\" (UID: \"87203750-2eb1-4b8c-9c94-b7ca88363b40\") " pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.714699 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.714504    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 17:04:21.714699 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.714600    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 17:04:21.714699 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.714666    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9ps2r\""
May 26 17:04:21.715303 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.715287    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 17:04:21.715628 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.715560    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 17:04:21.715628 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.715574    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 17:04:21.715759 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.715655    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m4ncc\""
May 26 17:04:21.723977 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.723958    2571 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 17:04:21.748227 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.748167    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 16:59:20 +0000 UTC" deadline="2028-03-11 09:59:24.617363046 +0000 UTC"
May 26 17:04:21.748227 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.748197    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15712h55m2.869169167s"
May 26 17:04:21.802933 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.802900    2571 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 17:04:21.814025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.813978    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8ce63184-976c-4e94-9b6c-c94a79abbf53-konnectivity-ca\") pod \"konnectivity-agent-r6jrs\" (UID: \"8ce63184-976c-4e94-9b6c-c94a79abbf53\") " pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:21.814194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814041    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-cni-bin\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.814194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814063    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-socket-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.814194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814080    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hdh6m\" (UniqueName: \"kubernetes.io/projected/2067d39e-24b4-4da4-9631-2f927333a9ef-kube-api-access-hdh6m\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.814194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814101    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-system-cni-dir\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.814194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-tuned\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.814194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814150    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-cni-netd\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.814194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814173    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-registration-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814197    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-sys-fs\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814232    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9c75f58-61b8-4762-a4b4-ceef1c52bf7b-host\") pod \"node-ca-6w2fz\" (UID: \"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b\") " pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814240    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-socket-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-multus-socket-dir-parent\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814285    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814304    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-cni-bin\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814313    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvt2p\" (UniqueName: \"kubernetes.io/projected/a13dbe45-e72d-4315-a2a1-c4736202fd9b-kube-api-access-kvt2p\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814340    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-slash\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814355    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-registration-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814366    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-var-lib-openvswitch\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814393    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-run-k8s-cni-cncf-io\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814419    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-device-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgcg\" (UniqueName: \"kubernetes.io/projected/f9c75f58-61b8-4762-a4b4-ceef1c52bf7b-kube-api-access-rcgcg\") pod \"node-ca-6w2fz\" (UID: \"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b\") " pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66cf\" (UniqueName: \"kubernetes.io/projected/0ace1a7d-bd84-4be0-8dc5-63f855978478-kube-api-access-t66cf\") pod \"iptables-alerter-b498j\" (UID: \"0ace1a7d-bd84-4be0-8dc5-63f855978478\") " pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-sysctl-conf\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-var-lib-kubelet\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.814558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814561    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87203750-2eb1-4b8c-9c94-b7ca88363b40-hosts-file\") pod \"node-resolver-m97tn\" (UID: \"87203750-2eb1-4b8c-9c94-b7ca88363b40\") " pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814585    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-etc-openvswitch\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814611    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814639    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-os-release\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814649    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8ce63184-976c-4e94-9b6c-c94a79abbf53-konnectivity-ca\") pod \"konnectivity-agent-r6jrs\" (UID: \"8ce63184-976c-4e94-9b6c-c94a79abbf53\") " pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814683    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-cni-netd\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814686    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b89e58ac-045c-4d32-951c-a0a00befc975-cni-binary-copy\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814736    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-lib-modules\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814736    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-device-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814806    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-slash\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814842    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814861    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-sys-fs\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814877    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-var-lib-openvswitch\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814906    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-run-netns\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814910    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87203750-2eb1-4b8c-9c94-b7ca88363b40-hosts-file\") pod \"node-resolver-m97tn\" (UID: \"87203750-2eb1-4b8c-9c94-b7ca88363b40\") " pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814931    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-etc-openvswitch\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814954    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.815363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.814965    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-run-netns\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815022    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815003    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-multus-cni-dir\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815073    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-sysconfig\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815102    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815138    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0ace1a7d-bd84-4be0-8dc5-63f855978478-iptables-alerter-script\") pod \"iptables-alerter-b498j\" (UID: \"0ace1a7d-bd84-4be0-8dc5-63f855978478\") " pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815179    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-var-lib-cni-multus\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815220    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8ce63184-976c-4e94-9b6c-c94a79abbf53-agent-certs\") pod \"konnectivity-agent-r6jrs\" (UID: \"8ce63184-976c-4e94-9b6c-c94a79abbf53\") " pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815253    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-node-log\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815281    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-multus-conf-dir\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815304    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-cnibin\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815331    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-log-socket\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815305    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-node-log\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815381    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2753c993-0102-4e57-b557-f03934174f89-ovnkube-config\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815412    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-log-socket\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815451    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ace1a7d-bd84-4be0-8dc5-63f855978478-host-slash\") pod \"iptables-alerter-b498j\" (UID: \"0ace1a7d-bd84-4be0-8dc5-63f855978478\") " pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ch9\" (UniqueName: \"kubernetes.io/projected/c9b5a4af-6d8b-411b-8339-65a20f76591c-kube-api-access-p5ch9\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.816102 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-var-lib-kubelet\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815559    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6d7\" (UniqueName: \"kubernetes.io/projected/87203750-2eb1-4b8c-9c94-b7ca88363b40-kube-api-access-fr6d7\") pod \"node-resolver-m97tn\" (UID: \"87203750-2eb1-4b8c-9c94-b7ca88363b40\") " pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815587    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-run-systemd\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815631    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815753    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-run-ovn\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815784    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9c75f58-61b8-4762-a4b4-ceef1c52bf7b-serviceca\") pod \"node-ca-6w2fz\" (UID: \"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b\") " pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815812    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b89e58ac-045c-4d32-951c-a0a00befc975-multus-daemon-config\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815836    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-modprobe-d\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815860    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-systemd\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815887    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-run-openvswitch\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815912    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-run-ovn-kubernetes\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815939    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-hostroot\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815965    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2753c993-0102-4e57-b557-f03934174f89-ovnkube-config\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.815962    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-run-multus-certs\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816004    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xrjx\" (UniqueName: \"kubernetes.io/projected/b89e58ac-045c-4d32-951c-a0a00befc975-kube-api-access-5xrjx\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816050    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-run-systemd\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-os-release\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816141    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-run-ovn\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.816855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816182    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-run-openvswitch\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816185    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-run-ovn-kubernetes\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816220    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:21.816309    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:21.816403    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs podName:a13dbe45-e72d-4315-a2a1-c4736202fd9b nodeName:}" failed. No retries permitted until 2026-05-26 17:04:22.316361022 +0000 UTC m=+3.041450509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs") pod "network-metrics-daemon-lztjw" (UID: "a13dbe45-e72d-4315-a2a1-c4736202fd9b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816498    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2753c993-0102-4e57-b557-f03934174f89-ovn-node-metrics-cert\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-run-netns\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816556    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-etc-kubernetes\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816578    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9b5a4af-6d8b-411b-8339-65a20f76591c-tmp\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a60644b9-b57b-4023-bc10-ddb686016835-cni-binary-copy\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816637    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a60644b9-b57b-4023-bc10-ddb686016835-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816664    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-kubelet\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816687    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2753c993-0102-4e57-b557-f03934174f89-env-overrides\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816710    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816773    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2067d39e-24b4-4da4-9631-2f927333a9ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816792    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-cnibin\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816836    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-host-kubelet\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.817695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-run\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816893    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-host\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816917    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68d6m\" (UniqueName: \"kubernetes.io/projected/a60644b9-b57b-4023-bc10-ddb686016835-kube-api-access-68d6m\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816951    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-systemd-units\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.816977    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-kubernetes\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-sys\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817046    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2753c993-0102-4e57-b557-f03934174f89-ovnkube-script-lib\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817107    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-sysctl-d\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817133    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-system-cni-dir\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817161    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87203750-2eb1-4b8c-9c94-b7ca88363b40-tmp-dir\") pod \"node-resolver-m97tn\" (UID: \"87203750-2eb1-4b8c-9c94-b7ca88363b40\") " pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817188    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9m8\" (UniqueName: \"kubernetes.io/projected/2753c993-0102-4e57-b557-f03934174f89-kube-api-access-5d9m8\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817211    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2753c993-0102-4e57-b557-f03934174f89-env-overrides\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817216    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-var-lib-cni-bin\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817246    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a60644b9-b57b-4023-bc10-ddb686016835-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817597    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87203750-2eb1-4b8c-9c94-b7ca88363b40-tmp-dir\") pod \"node-resolver-m97tn\" (UID: \"87203750-2eb1-4b8c-9c94-b7ca88363b40\") " pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817662    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2753c993-0102-4e57-b557-f03934174f89-systemd-units\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.818482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.817663    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2753c993-0102-4e57-b557-f03934174f89-ovnkube-script-lib\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.820605 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.820569    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8ce63184-976c-4e94-9b6c-c94a79abbf53-agent-certs\") pod \"konnectivity-agent-r6jrs\" (UID: \"8ce63184-976c-4e94-9b6c-c94a79abbf53\") " pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:21.822576 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.822552    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdh6m\" (UniqueName: \"kubernetes.io/projected/2067d39e-24b4-4da4-9631-2f927333a9ef-kube-api-access-hdh6m\") pod \"aws-ebs-csi-driver-node-dvh4h\" (UID: \"2067d39e-24b4-4da4-9631-2f927333a9ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:21.823468 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.822894    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6d7\" (UniqueName: \"kubernetes.io/projected/87203750-2eb1-4b8c-9c94-b7ca88363b40-kube-api-access-fr6d7\") pod \"node-resolver-m97tn\" (UID: \"87203750-2eb1-4b8c-9c94-b7ca88363b40\") " pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:21.823468 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.823301    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvt2p\" (UniqueName: \"kubernetes.io/projected/a13dbe45-e72d-4315-a2a1-c4736202fd9b-kube-api-access-kvt2p\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:21.828281 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.828262    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2753c993-0102-4e57-b557-f03934174f89-ovn-node-metrics-cert\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.830295 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.830248    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9m8\" (UniqueName: \"kubernetes.io/projected/2753c993-0102-4e57-b557-f03934174f89-kube-api-access-5d9m8\") pod \"ovnkube-node-r56x2\" (UID: \"2753c993-0102-4e57-b557-f03934174f89\") " pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:21.833835 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.833799    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal" event={"ID":"f09d26b0580610a005e903ab2681d50d","Type":"ContainerStarted","Data":"76df480db343cbe4a113ca87c069241229e2eeafc1cae3002fcb4be836697f9e"}
May 26 17:04:21.834817 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.834795    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal" event={"ID":"3a9722eed112a49f65691e180ade29ba","Type":"ContainerStarted","Data":"9a8ea5394ddc47026b6d17fa5d96895171b7ff2fbfb1d9c3f1ebd1a3be87f6f3"}
May 26 17:04:21.918049 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.917993    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9c75f58-61b8-4762-a4b4-ceef1c52bf7b-serviceca\") pod \"node-ca-6w2fz\" (UID: \"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b\") " pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.918049 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918055    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b89e58ac-045c-4d32-951c-a0a00befc975-multus-daemon-config\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918078    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-modprobe-d\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918093    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-systemd\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918114    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-hostroot\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918135    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-run-multus-certs\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918158    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5xrjx\" (UniqueName: \"kubernetes.io/projected/b89e58ac-045c-4d32-951c-a0a00befc975-kube-api-access-5xrjx\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918179    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-os-release\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918203    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-hostroot\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918232    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-modprobe-d\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918248    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-run-netns\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918203    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-systemd\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918216    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-run-netns\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918294    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-etc-kubernetes\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918320    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9b5a4af-6d8b-411b-8339-65a20f76591c-tmp\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918341    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-os-release\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918346    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a60644b9-b57b-4023-bc10-ddb686016835-cni-binary-copy\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918373    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a60644b9-b57b-4023-bc10-ddb686016835-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918378    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-etc-kubernetes\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918403    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-cnibin\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918415    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-run-multus-certs\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918427    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-run\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918451    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-host\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918458    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9c75f58-61b8-4762-a4b4-ceef1c52bf7b-serviceca\") pod \"node-ca-6w2fz\" (UID: \"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b\") " pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918474    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-68d6m\" (UniqueName: \"kubernetes.io/projected/a60644b9-b57b-4023-bc10-ddb686016835-kube-api-access-68d6m\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918514    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-kubernetes\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918540    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-sys\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918563    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-sysctl-d\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918607    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-system-cni-dir\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918620    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-host\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.918770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918635    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-var-lib-cni-bin\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918659    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a60644b9-b57b-4023-bc10-ddb686016835-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918675    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-kubernetes\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918710    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-system-cni-dir\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918749    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-tuned\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918788    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-sysctl-d\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918797    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b89e58ac-045c-4d32-951c-a0a00befc975-multus-daemon-config\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918814    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-sys\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918859    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9c75f58-61b8-4762-a4b4-ceef1c52bf7b-host\") pod \"node-ca-6w2fz\" (UID: \"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b\") " pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918889    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-multus-socket-dir-parent\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918897    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-cnibin\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918895    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-system-cni-dir\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918860    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-system-cni-dir\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918931    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-var-lib-cni-bin\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918946    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9c75f58-61b8-4762-a4b4-ceef1c52bf7b-host\") pod \"node-ca-6w2fz\" (UID: \"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b\") " pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918953    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-run\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.918971    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919051    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-multus-socket-dir-parent\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.919554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919057    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a60644b9-b57b-4023-bc10-ddb686016835-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919092    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-run-k8s-cni-cncf-io\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919122    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgcg\" (UniqueName: \"kubernetes.io/projected/f9c75f58-61b8-4762-a4b4-ceef1c52bf7b-kube-api-access-rcgcg\") pod \"node-ca-6w2fz\" (UID: \"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b\") " pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919151    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t66cf\" (UniqueName: \"kubernetes.io/projected/0ace1a7d-bd84-4be0-8dc5-63f855978478-kube-api-access-t66cf\") pod \"iptables-alerter-b498j\" (UID: \"0ace1a7d-bd84-4be0-8dc5-63f855978478\") " pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919161    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-run-k8s-cni-cncf-io\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919177    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-sysctl-conf\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919205    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-var-lib-kubelet\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919232    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-os-release\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919258    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b89e58ac-045c-4d32-951c-a0a00befc975-cni-binary-copy\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919283    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-lib-modules\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919309    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-multus-cni-dir\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919334    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-sysconfig\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919365    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919370    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a60644b9-b57b-4023-bc10-ddb686016835-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919374    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-os-release\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919378    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-sysctl-conf\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919392    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0ace1a7d-bd84-4be0-8dc5-63f855978478-iptables-alerter-script\") pod \"iptables-alerter-b498j\" (UID: \"0ace1a7d-bd84-4be0-8dc5-63f855978478\") " pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.920254 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919477    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-lib-modules\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919507    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-var-lib-kubelet\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919533    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919557    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a60644b9-b57b-4023-bc10-ddb686016835-cni-binary-copy\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919601    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-sysconfig\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919648    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-var-lib-cni-multus\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919659    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-multus-cni-dir\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919685    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-multus-conf-dir\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919704    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-var-lib-cni-multus\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919712    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-cnibin\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919739    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ace1a7d-bd84-4be0-8dc5-63f855978478-host-slash\") pod \"iptables-alerter-b498j\" (UID: \"0ace1a7d-bd84-4be0-8dc5-63f855978478\") " pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919749    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-multus-conf-dir\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919765    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ch9\" (UniqueName: \"kubernetes.io/projected/c9b5a4af-6d8b-411b-8339-65a20f76591c-kube-api-access-p5ch9\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919792    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-var-lib-kubelet\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919811    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ace1a7d-bd84-4be0-8dc5-63f855978478-host-slash\") pod \"iptables-alerter-b498j\" (UID: \"0ace1a7d-bd84-4be0-8dc5-63f855978478\") " pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919766    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a60644b9-b57b-4023-bc10-ddb686016835-cnibin\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919858    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b89e58ac-045c-4d32-951c-a0a00befc975-host-var-lib-kubelet\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919897    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0ace1a7d-bd84-4be0-8dc5-63f855978478-iptables-alerter-script\") pod \"iptables-alerter-b498j\" (UID: \"0ace1a7d-bd84-4be0-8dc5-63f855978478\") " pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.920701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.919903    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b89e58ac-045c-4d32-951c-a0a00befc975-cni-binary-copy\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.921463 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.921443    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9b5a4af-6d8b-411b-8339-65a20f76591c-etc-tuned\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.921618 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.921603    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9b5a4af-6d8b-411b-8339-65a20f76591c-tmp\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.926848 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:21.926546    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 17:04:21.926848 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:21.926574    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 17:04:21.926848 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:21.926587    2571 projected.go:194] Error preparing data for projected volume kube-api-access-657zs for pod openshift-network-diagnostics/network-check-target-fjgrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:21.926848 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:21.926668    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs podName:9bd326f5-4fac-48da-b113-83b2234bc506 nodeName:}" failed. No retries permitted until 2026-05-26 17:04:22.426650656 +0000 UTC m=+3.151740152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-657zs" (UniqueName: "kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs") pod "network-check-target-fjgrn" (UID: "9bd326f5-4fac-48da-b113-83b2234bc506") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:21.929115 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.929093    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgcg\" (UniqueName: \"kubernetes.io/projected/f9c75f58-61b8-4762-a4b4-ceef1c52bf7b-kube-api-access-rcgcg\") pod \"node-ca-6w2fz\" (UID: \"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b\") " pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:21.929233 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.929185    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ch9\" (UniqueName: \"kubernetes.io/projected/c9b5a4af-6d8b-411b-8339-65a20f76591c-kube-api-access-p5ch9\") pod \"tuned-bz4lz\" (UID: \"c9b5a4af-6d8b-411b-8339-65a20f76591c\") " pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:21.929233 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.929197    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66cf\" (UniqueName: \"kubernetes.io/projected/0ace1a7d-bd84-4be0-8dc5-63f855978478-kube-api-access-t66cf\") pod \"iptables-alerter-b498j\" (UID: \"0ace1a7d-bd84-4be0-8dc5-63f855978478\") " pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:21.930171 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.930098    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xrjx\" (UniqueName: \"kubernetes.io/projected/b89e58ac-045c-4d32-951c-a0a00befc975-kube-api-access-5xrjx\") pod \"multus-xrkfv\" (UID: \"b89e58ac-045c-4d32-951c-a0a00befc975\") " pod="openshift-multus/multus-xrkfv"
May 26 17:04:21.930171 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:21.930153    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68d6m\" (UniqueName: \"kubernetes.io/projected/a60644b9-b57b-4023-bc10-ddb686016835-kube-api-access-68d6m\") pod \"multus-additional-cni-plugins-8qh7j\" (UID: \"a60644b9-b57b-4023-bc10-ddb686016835\") " pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:22.009723 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.009681    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:22.018595 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.018571    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:22.030321 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.030295    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h"
May 26 17:04:22.033814 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.033797    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m97tn"
May 26 17:04:22.042438 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.042416    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xrkfv"
May 26 17:04:22.050975 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.050957    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bz4lz"
May 26 17:04:22.058553 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.058532    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6w2fz"
May 26 17:04:22.067095 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.067077    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8qh7j"
May 26 17:04:22.072540 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.072520    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b498j"
May 26 17:04:22.322387 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.322347    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:22.322580 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:22.322482    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:22.322580 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:22.322556    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs podName:a13dbe45-e72d-4315-a2a1-c4736202fd9b nodeName:}" failed. No retries permitted until 2026-05-26 17:04:23.32253448 +0000 UTC m=+4.047623972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs") pod "network-metrics-daemon-lztjw" (UID: "a13dbe45-e72d-4315-a2a1-c4736202fd9b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:22.413102 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:22.412950    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce63184_976c_4e94_9b6c_c94a79abbf53.slice/crio-ffd3c69329406a01bc700b2f9c4c6c3c78026538f4af5b804dc9564f40b3d2b6 WatchSource:0}: Error finding container ffd3c69329406a01bc700b2f9c4c6c3c78026538f4af5b804dc9564f40b3d2b6: Status 404 returned error can't find the container with id ffd3c69329406a01bc700b2f9c4c6c3c78026538f4af5b804dc9564f40b3d2b6
May 26 17:04:22.414736 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:22.414716    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b5a4af_6d8b_411b_8339_65a20f76591c.slice/crio-f4bc610651830d04259d2bda3385b9c4eb1e4a81a76ed044d3476f2c712bfd5e WatchSource:0}: Error finding container f4bc610651830d04259d2bda3385b9c4eb1e4a81a76ed044d3476f2c712bfd5e: Status 404 returned error can't find the container with id f4bc610651830d04259d2bda3385b9c4eb1e4a81a76ed044d3476f2c712bfd5e
May 26 17:04:22.416854 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:22.416826    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ace1a7d_bd84_4be0_8dc5_63f855978478.slice/crio-dc91cd5e027f925fc9a89d1cd086458af1703b86383f3923152a9ffed558bb0d WatchSource:0}: Error finding container dc91cd5e027f925fc9a89d1cd086458af1703b86383f3923152a9ffed558bb0d: Status 404 returned error can't find the container with id dc91cd5e027f925fc9a89d1cd086458af1703b86383f3923152a9ffed558bb0d
May 26 17:04:22.418915 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:22.418872    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c75f58_61b8_4762_a4b4_ceef1c52bf7b.slice/crio-c44c1908fcdbff855c9a034221ba4913d734465fb9e1271fb2172ba24c7531cc WatchSource:0}: Error finding container c44c1908fcdbff855c9a034221ba4913d734465fb9e1271fb2172ba24c7531cc: Status 404 returned error can't find the container with id c44c1908fcdbff855c9a034221ba4913d734465fb9e1271fb2172ba24c7531cc
May 26 17:04:22.419684 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:22.419646    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60644b9_b57b_4023_bc10_ddb686016835.slice/crio-296d1a3564aedf6135413e29ef5a35ed67b51ec42b762fb081e4e4bf49284227 WatchSource:0}: Error finding container 296d1a3564aedf6135413e29ef5a35ed67b51ec42b762fb081e4e4bf49284227: Status 404 returned error can't find the container with id 296d1a3564aedf6135413e29ef5a35ed67b51ec42b762fb081e4e4bf49284227
May 26 17:04:22.421441 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:22.421217    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87203750_2eb1_4b8c_9c94_b7ca88363b40.slice/crio-7a793d5f36c90c612e6ea6d2c42f1ba0c7bc33b972f976a63ab67fce2d76920a WatchSource:0}: Error finding container 7a793d5f36c90c612e6ea6d2c42f1ba0c7bc33b972f976a63ab67fce2d76920a: Status 404 returned error can't find the container with id 7a793d5f36c90c612e6ea6d2c42f1ba0c7bc33b972f976a63ab67fce2d76920a
May 26 17:04:22.423030 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:22.422976    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89e58ac_045c_4d32_951c_a0a00befc975.slice/crio-829135e545ed3fed78d76e19a0248dd6706a7cde09cb34d1bd81592ca53a2ed8 WatchSource:0}: Error finding container 829135e545ed3fed78d76e19a0248dd6706a7cde09cb34d1bd81592ca53a2ed8: Status 404 returned error can't find the container with id 829135e545ed3fed78d76e19a0248dd6706a7cde09cb34d1bd81592ca53a2ed8
May 26 17:04:22.423263 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:22.423244    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2753c993_0102_4e57_b557_f03934174f89.slice/crio-79df25069c0247fd88c2fc474c199c8ad6763fa4c53897220f917f3397897164 WatchSource:0}: Error finding container 79df25069c0247fd88c2fc474c199c8ad6763fa4c53897220f917f3397897164: Status 404 returned error can't find the container with id 79df25069c0247fd88c2fc474c199c8ad6763fa4c53897220f917f3397897164
May 26 17:04:22.424313 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:04:22.424275    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2067d39e_24b4_4da4_9631_2f927333a9ef.slice/crio-62977960119e12200e06d2632f0e1a756d007bfe02f151de60d4475408f66094 WatchSource:0}: Error finding container 62977960119e12200e06d2632f0e1a756d007bfe02f151de60d4475408f66094: Status 404 returned error can't find the container with id 62977960119e12200e06d2632f0e1a756d007bfe02f151de60d4475408f66094
May 26 17:04:22.523725 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.523692    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:22.523859 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:22.523840    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 17:04:22.523907 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:22.523864    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 17:04:22.523907 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:22.523874    2571 projected.go:194] Error preparing data for projected volume kube-api-access-657zs for pod openshift-network-diagnostics/network-check-target-fjgrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:22.523972 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:22.523922    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs podName:9bd326f5-4fac-48da-b113-83b2234bc506 nodeName:}" failed. No retries permitted until 2026-05-26 17:04:23.523907844 +0000 UTC m=+4.248997324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-657zs" (UniqueName: "kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs") pod "network-check-target-fjgrn" (UID: "9bd326f5-4fac-48da-b113-83b2234bc506") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:22.748808 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.748768    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 16:59:20 +0000 UTC" deadline="2027-11-29 01:28:35.336955533 +0000 UTC"
May 26 17:04:22.748808 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.748803    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13232h24m12.588155025s"
May 26 17:04:22.830436 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.830393    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:22.830591 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:22.830553    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:22.849586 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.849550    2571 generic.go:358] "Generic (PLEG): container finished" podID="3a9722eed112a49f65691e180ade29ba" containerID="0fb16439896f7e14d3bdfd03fd9da26e18146522428a568e85a9ff18448d3837" exitCode=0
May 26 17:04:22.849727 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.849630    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal" event={"ID":"3a9722eed112a49f65691e180ade29ba","Type":"ContainerDied","Data":"0fb16439896f7e14d3bdfd03fd9da26e18146522428a568e85a9ff18448d3837"}
May 26 17:04:22.860053 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.860003    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerStarted","Data":"79df25069c0247fd88c2fc474c199c8ad6763fa4c53897220f917f3397897164"}
May 26 17:04:22.863048 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.863021    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" event={"ID":"a60644b9-b57b-4023-bc10-ddb686016835","Type":"ContainerStarted","Data":"296d1a3564aedf6135413e29ef5a35ed67b51ec42b762fb081e4e4bf49284227"}
May 26 17:04:22.870407 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.870383    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b498j" event={"ID":"0ace1a7d-bd84-4be0-8dc5-63f855978478","Type":"ContainerStarted","Data":"dc91cd5e027f925fc9a89d1cd086458af1703b86383f3923152a9ffed558bb0d"}
May 26 17:04:22.878138 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.878111    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bz4lz" event={"ID":"c9b5a4af-6d8b-411b-8339-65a20f76591c","Type":"ContainerStarted","Data":"f4bc610651830d04259d2bda3385b9c4eb1e4a81a76ed044d3476f2c712bfd5e"}
May 26 17:04:22.881614 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.881588    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h" event={"ID":"2067d39e-24b4-4da4-9631-2f927333a9ef","Type":"ContainerStarted","Data":"62977960119e12200e06d2632f0e1a756d007bfe02f151de60d4475408f66094"}
May 26 17:04:22.885357 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.885333    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xrkfv" event={"ID":"b89e58ac-045c-4d32-951c-a0a00befc975","Type":"ContainerStarted","Data":"829135e545ed3fed78d76e19a0248dd6706a7cde09cb34d1bd81592ca53a2ed8"}
May 26 17:04:22.887709 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.887682    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m97tn" event={"ID":"87203750-2eb1-4b8c-9c94-b7ca88363b40","Type":"ContainerStarted","Data":"7a793d5f36c90c612e6ea6d2c42f1ba0c7bc33b972f976a63ab67fce2d76920a"}
May 26 17:04:22.890786 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.890763    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6w2fz" event={"ID":"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b","Type":"ContainerStarted","Data":"c44c1908fcdbff855c9a034221ba4913d734465fb9e1271fb2172ba24c7531cc"}
May 26 17:04:22.899671 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.899640    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r6jrs" event={"ID":"8ce63184-976c-4e94-9b6c-c94a79abbf53","Type":"ContainerStarted","Data":"ffd3c69329406a01bc700b2f9c4c6c3c78026538f4af5b804dc9564f40b3d2b6"}
May 26 17:04:22.906596 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.906570    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal" event={"ID":"f09d26b0580610a005e903ab2681d50d","Type":"ContainerStarted","Data":"146dd42d933e4f359b45e254d1f859a801d2a94867b972cfbca2a0742235926c"}
May 26 17:04:22.922060 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:22.921952    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-109.ec2.internal" podStartSLOduration=1.921936426 podStartE2EDuration="1.921936426s" podCreationTimestamp="2026-05-26 17:04:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:04:22.921427991 +0000 UTC m=+3.646517495" watchObservedRunningTime="2026-05-26 17:04:22.921936426 +0000 UTC m=+3.647025927"
May 26 17:04:23.328168 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:23.328130    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:23.328340 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:23.328325    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:23.328414 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:23.328389    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs podName:a13dbe45-e72d-4315-a2a1-c4736202fd9b nodeName:}" failed. No retries permitted until 2026-05-26 17:04:25.328370123 +0000 UTC m=+6.053459607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs") pod "network-metrics-daemon-lztjw" (UID: "a13dbe45-e72d-4315-a2a1-c4736202fd9b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:23.529718 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:23.529653    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:23.529979 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:23.529961    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 17:04:23.530084 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:23.529983    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 17:04:23.530084 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:23.529996    2571 projected.go:194] Error preparing data for projected volume kube-api-access-657zs for pod openshift-network-diagnostics/network-check-target-fjgrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:23.530084 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:23.530073    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs podName:9bd326f5-4fac-48da-b113-83b2234bc506 nodeName:}" failed. No retries permitted until 2026-05-26 17:04:25.530053603 +0000 UTC m=+6.255143094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-657zs" (UniqueName: "kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs") pod "network-check-target-fjgrn" (UID: "9bd326f5-4fac-48da-b113-83b2234bc506") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:23.844828 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:23.844139    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:23.844828 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:23.844319    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:23.927100 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:23.927065    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal" event={"ID":"3a9722eed112a49f65691e180ade29ba","Type":"ContainerStarted","Data":"80ae6abae94f52b2df6804aefd34f2f38efabd07b7057f6f5b08e2384b923b61"}
May 26 17:04:24.830192 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:24.830157    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:24.830391 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:24.830293    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:25.356551 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:25.355897    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:25.356551 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:25.356109    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:25.356551 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:25.356171    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs podName:a13dbe45-e72d-4315-a2a1-c4736202fd9b nodeName:}" failed. No retries permitted until 2026-05-26 17:04:29.356152204 +0000 UTC m=+10.081241700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs") pod "network-metrics-daemon-lztjw" (UID: "a13dbe45-e72d-4315-a2a1-c4736202fd9b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:25.557757 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:25.557508    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:25.557918 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:25.557695    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 17:04:25.557918 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:25.557855    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 17:04:25.557918 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:25.557873    2571 projected.go:194] Error preparing data for projected volume kube-api-access-657zs for pod openshift-network-diagnostics/network-check-target-fjgrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:25.558123 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:25.557940    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs podName:9bd326f5-4fac-48da-b113-83b2234bc506 nodeName:}" failed. No retries permitted until 2026-05-26 17:04:29.557917899 +0000 UTC m=+10.283007396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-657zs" (UniqueName: "kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs") pod "network-check-target-fjgrn" (UID: "9bd326f5-4fac-48da-b113-83b2234bc506") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:25.831200 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:25.831168    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:25.831371 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:25.831295    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:26.830481 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:26.830421    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:26.830946 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:26.830573    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:27.830884 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:27.830628    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:27.830884 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:27.830746    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:28.830795 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:28.830633    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:28.830795 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:28.830778    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:29.389005 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:29.388967    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:29.389188 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:29.389135    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:29.389251 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:29.389204    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs podName:a13dbe45-e72d-4315-a2a1-c4736202fd9b nodeName:}" failed. No retries permitted until 2026-05-26 17:04:37.389183674 +0000 UTC m=+18.114273155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs") pod "network-metrics-daemon-lztjw" (UID: "a13dbe45-e72d-4315-a2a1-c4736202fd9b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:29.590412 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:29.590249    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:29.590595 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:29.590451    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 17:04:29.590595 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:29.590471    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 17:04:29.590595 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:29.590484    2571 projected.go:194] Error preparing data for projected volume kube-api-access-657zs for pod openshift-network-diagnostics/network-check-target-fjgrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:29.590595 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:29.590538    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs podName:9bd326f5-4fac-48da-b113-83b2234bc506 nodeName:}" failed. No retries permitted until 2026-05-26 17:04:37.590521514 +0000 UTC m=+18.315611007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-657zs" (UniqueName: "kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs") pod "network-check-target-fjgrn" (UID: "9bd326f5-4fac-48da-b113-83b2234bc506") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:29.831444 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:29.830928    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:29.831444 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:29.831061    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:30.830715 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:30.830676    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:30.830901 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:30.830868    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:31.830405 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:31.830369    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:31.830856 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:31.830498    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:32.830702 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:32.830658    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:32.831249 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:32.830818    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:33.829970 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:33.829933    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:33.830143 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:33.830085    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:34.830584 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:34.830540    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:34.831040 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:34.830682    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:35.830383 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:35.830338    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:35.830567 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:35.830449    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:36.830392 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:36.830359    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:36.830884 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:36.830487    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:37.451851 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:37.451811    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:37.452048 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:37.451973    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:37.452124 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:37.452063    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs podName:a13dbe45-e72d-4315-a2a1-c4736202fd9b nodeName:}" failed. No retries permitted until 2026-05-26 17:04:53.452044255 +0000 UTC m=+34.177133755 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs") pod "network-metrics-daemon-lztjw" (UID: "a13dbe45-e72d-4315-a2a1-c4736202fd9b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:37.653698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:37.653644    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:37.653867 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:37.653843    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 17:04:37.653907 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:37.653871    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 17:04:37.653907 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:37.653886    2571 projected.go:194] Error preparing data for projected volume kube-api-access-657zs for pod openshift-network-diagnostics/network-check-target-fjgrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:37.653985 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:37.653962    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs podName:9bd326f5-4fac-48da-b113-83b2234bc506 nodeName:}" failed. No retries permitted until 2026-05-26 17:04:53.653942507 +0000 UTC m=+34.379032002 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-657zs" (UniqueName: "kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs") pod "network-check-target-fjgrn" (UID: "9bd326f5-4fac-48da-b113-83b2234bc506") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:37.830168 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:37.830125    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:37.830344 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:37.830272    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:38.830723 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:38.830691    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:38.831320 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:38.830829    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:39.831117 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:39.831077    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:39.831581 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:39.831185    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:40.830180 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:40.830107    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:40.830313 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:40.830227    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:40.959982 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:40.959556    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bz4lz" event={"ID":"c9b5a4af-6d8b-411b-8339-65a20f76591c","Type":"ContainerStarted","Data":"8e067dd6d04a0dfd4e245acf8281e188f76d226fb26bd7997217538e087b72f7"}
May 26 17:04:40.961651 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:40.961622    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h" event={"ID":"2067d39e-24b4-4da4-9631-2f927333a9ef","Type":"ContainerStarted","Data":"06f53af78f1a16cc40f1ef859dacc3d9838d12c2dd90b79f44d537afbee1c38e"}
May 26 17:04:40.963248 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:40.963220    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xrkfv" event={"ID":"b89e58ac-045c-4d32-951c-a0a00befc975","Type":"ContainerStarted","Data":"7bbff7c883a7257ae00592e4bd4305c93448dd1ff8e0ca304e591d799679b71c"}
May 26 17:04:40.969125 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:40.969080    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r6jrs" event={"ID":"8ce63184-976c-4e94-9b6c-c94a79abbf53","Type":"ContainerStarted","Data":"6c4ac2f9770f461487bb1bfc27fab0077d74f1e8d3e610cdaa5966a1d7e1044c"}
May 26 17:04:40.984195 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:40.984044    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-109.ec2.internal" podStartSLOduration=19.984031226 podStartE2EDuration="19.984031226s" podCreationTimestamp="2026-05-26 17:04:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:04:23.946726388 +0000 UTC m=+4.671815889" watchObservedRunningTime="2026-05-26 17:04:40.984031226 +0000 UTC m=+21.709120726"
May 26 17:04:40.984357 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:40.984228    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bz4lz" podStartSLOduration=2.830696088 podStartE2EDuration="20.984222567s" podCreationTimestamp="2026-05-26 17:04:20 +0000 UTC" firstStartedPulling="2026-05-26 17:04:22.416166858 +0000 UTC m=+3.141256337" lastFinishedPulling="2026-05-26 17:04:40.569693325 +0000 UTC m=+21.294782816" observedRunningTime="2026-05-26 17:04:40.983750928 +0000 UTC m=+21.708840429" watchObservedRunningTime="2026-05-26 17:04:40.984222567 +0000 UTC m=+21.709312070"
May 26 17:04:41.001635 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.000315    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r6jrs" podStartSLOduration=3.866229467 podStartE2EDuration="22.000296367s" podCreationTimestamp="2026-05-26 17:04:19 +0000 UTC" firstStartedPulling="2026-05-26 17:04:22.414801164 +0000 UTC m=+3.139890643" lastFinishedPulling="2026-05-26 17:04:40.548868051 +0000 UTC m=+21.273957543" observedRunningTime="2026-05-26 17:04:40.99898417 +0000 UTC m=+21.724073671" watchObservedRunningTime="2026-05-26 17:04:41.000296367 +0000 UTC m=+21.725385887"
May 26 17:04:41.015962 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.015906    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-xrkfv" podStartSLOduration=2.859739639 podStartE2EDuration="21.015892257s" podCreationTimestamp="2026-05-26 17:04:20 +0000 UTC" firstStartedPulling="2026-05-26 17:04:22.425663889 +0000 UTC m=+3.150753369" lastFinishedPulling="2026-05-26 17:04:40.581816493 +0000 UTC m=+21.306905987" observedRunningTime="2026-05-26 17:04:41.015589656 +0000 UTC m=+21.740679160" watchObservedRunningTime="2026-05-26 17:04:41.015892257 +0000 UTC m=+21.740981758"
May 26 17:04:41.831043 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.830807    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:41.831180 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:41.831159    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:41.973713 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.973678    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m97tn" event={"ID":"87203750-2eb1-4b8c-9c94-b7ca88363b40","Type":"ContainerStarted","Data":"9c144e1c98d5ea5fdd509eac089dea72f4481f7054add48b93ee8202eacd86d8"}
May 26 17:04:41.975147 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.975119    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6w2fz" event={"ID":"f9c75f58-61b8-4762-a4b4-ceef1c52bf7b","Type":"ContainerStarted","Data":"8dcbc3e2203082b43817938066697d38c96ce15b5738b0d301d1cb351d2b266c"}
May 26 17:04:41.977668 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.977645    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:04:41.977972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.977945    2571 generic.go:358] "Generic (PLEG): container finished" podID="2753c993-0102-4e57-b557-f03934174f89" containerID="6200c17fa1e952b4ddd5995db447d1121fbe986bd41450302b081a7dc0b60618" exitCode=1
May 26 17:04:41.977972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.977962    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerStarted","Data":"22d8b6718b7c7044b8d659623b969ca11841cd3b15e0ab65f367f788b563d11d"}
May 26 17:04:41.978147 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.977988    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerStarted","Data":"3cdba26a10e3c9266051456e75beefbb688c15fbe12bd92f6a5305be9b0ef55b"}
May 26 17:04:41.978147 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.977998    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerStarted","Data":"27aa2cee71ddf7ed27c5a5a5f0870ba20b96a473fc85f2098e7ed509e6906c59"}
May 26 17:04:41.978147 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.978023    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerStarted","Data":"1a9c7c76a11d3a18e1c17e4349c6dd322196459a589eea20751e1e04ec513988"}
May 26 17:04:41.978147 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.978035    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerDied","Data":"6200c17fa1e952b4ddd5995db447d1121fbe986bd41450302b081a7dc0b60618"}
May 26 17:04:41.978147 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.978049    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerStarted","Data":"e29864ac84ab2117a7616188c92b9f4111a74405db117e8929c2f70209156c5a"}
May 26 17:04:41.979290 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.979267    2571 generic.go:358] "Generic (PLEG): container finished" podID="a60644b9-b57b-4023-bc10-ddb686016835" containerID="5b6d5bbb8ed243e53d6421e78f729ab1957f0d3d2eb5942b1e157fa08e61075a" exitCode=0
May 26 17:04:41.979384 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.979368    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" event={"ID":"a60644b9-b57b-4023-bc10-ddb686016835","Type":"ContainerDied","Data":"5b6d5bbb8ed243e53d6421e78f729ab1957f0d3d2eb5942b1e157fa08e61075a"}
May 26 17:04:41.987595 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:41.987552    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-m97tn" podStartSLOduration=3.835551486 podStartE2EDuration="21.987539178s" podCreationTimestamp="2026-05-26 17:04:20 +0000 UTC" firstStartedPulling="2026-05-26 17:04:22.423795376 +0000 UTC m=+3.148884861" lastFinishedPulling="2026-05-26 17:04:40.57578307 +0000 UTC m=+21.300872553" observedRunningTime="2026-05-26 17:04:41.987115939 +0000 UTC m=+22.712205440" watchObservedRunningTime="2026-05-26 17:04:41.987539178 +0000 UTC m=+22.712628679"
May 26 17:04:42.000721 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.000662    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6w2fz" podStartSLOduration=3.85177794 podStartE2EDuration="22.000644725s" podCreationTimestamp="2026-05-26 17:04:20 +0000 UTC" firstStartedPulling="2026-05-26 17:04:22.420825944 +0000 UTC m=+3.145915427" lastFinishedPulling="2026-05-26 17:04:40.569692719 +0000 UTC m=+21.294782212" observedRunningTime="2026-05-26 17:04:42.000437809 +0000 UTC m=+22.725527332" watchObservedRunningTime="2026-05-26 17:04:42.000644725 +0000 UTC m=+22.725734246"
May 26 17:04:42.117959 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.117929    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 17:04:42.186164 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.186130    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:42.186694 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.186678    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:42.783557 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.783433    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T17:04:42.117956851Z","UUID":"149443bc-ffdb-428c-8578-cf7d37594263","Handler":null,"Name":"","Endpoint":""}
May 26 17:04:42.786160 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.786132    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 17:04:42.786301 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.786169    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 17:04:42.830175 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.830144    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:42.830353 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:42.830287    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:42.982954 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.982909    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b498j" event={"ID":"0ace1a7d-bd84-4be0-8dc5-63f855978478","Type":"ContainerStarted","Data":"7a59679ab9dd7957432065386a10a0847211778c7111d9b51fcce643f1ec45f7"}
May 26 17:04:42.985251 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.985122    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h" event={"ID":"2067d39e-24b4-4da4-9631-2f927333a9ef","Type":"ContainerStarted","Data":"0e81b2f661603b058be298b11deca825e305cb1610030991feaf8208c4debc1b"}
May 26 17:04:42.985518 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.985498    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:42.986031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.985996    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r6jrs"
May 26 17:04:42.998438 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:42.998390    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b498j" podStartSLOduration=4.847295959 podStartE2EDuration="22.998376065s" podCreationTimestamp="2026-05-26 17:04:20 +0000 UTC" firstStartedPulling="2026-05-26 17:04:22.418614172 +0000 UTC m=+3.143703651" lastFinishedPulling="2026-05-26 17:04:40.569694263 +0000 UTC m=+21.294783757" observedRunningTime="2026-05-26 17:04:42.99760065 +0000 UTC m=+23.722690153" watchObservedRunningTime="2026-05-26 17:04:42.998376065 +0000 UTC m=+23.723465568"
May 26 17:04:43.830453 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:43.830418    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:43.830640 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:43.830538    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:43.989868 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:43.989766    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h" event={"ID":"2067d39e-24b4-4da4-9631-2f927333a9ef","Type":"ContainerStarted","Data":"37cccfa31b93f31096dfeb37df224ac160a3b1988edd052c981d39f9e6990d84"}
May 26 17:04:44.010509 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:44.010457    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvh4h" podStartSLOduration=4.090889767 podStartE2EDuration="25.010443375s" podCreationTimestamp="2026-05-26 17:04:19 +0000 UTC" firstStartedPulling="2026-05-26 17:04:22.426537514 +0000 UTC m=+3.151626992" lastFinishedPulling="2026-05-26 17:04:43.346091113 +0000 UTC m=+24.071180600" observedRunningTime="2026-05-26 17:04:44.007661827 +0000 UTC m=+24.732751323" watchObservedRunningTime="2026-05-26 17:04:44.010443375 +0000 UTC m=+24.735532875"
May 26 17:04:44.830732 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:44.830705    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:44.830914 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:44.830814    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:44.994719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:44.994694    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:04:44.995327 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:44.995105    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerStarted","Data":"e17ce53a45d8ecc69d70322823aa68706134b5622e225a3eac5bccdac3489caa"}
May 26 17:04:45.830984 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:45.830953    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:45.831190 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:45.831092    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:46.830398 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:46.830222    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:46.830827 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:46.830485    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:47.001546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:47.001520    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:04:47.001860 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:47.001836    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerStarted","Data":"9806d9b7ce40c2db23bbf92ffe6b1a7c8a2db94d72ba5186739c0a36e22ceb7d"}
May 26 17:04:47.002154 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:47.002138    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:47.002154 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:47.002164    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:47.002341 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:47.002320    2571 scope.go:117] "RemoveContainer" containerID="6200c17fa1e952b4ddd5995db447d1121fbe986bd41450302b081a7dc0b60618"
May 26 17:04:47.003671 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:47.003649    2571 generic.go:358] "Generic (PLEG): container finished" podID="a60644b9-b57b-4023-bc10-ddb686016835" containerID="a83758c165b41a6ededfe9b80bc8a919dd76ac6b692c47d89c5d1ecc888ad794" exitCode=0
May 26 17:04:47.003743 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:47.003702    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" event={"ID":"a60644b9-b57b-4023-bc10-ddb686016835","Type":"ContainerDied","Data":"a83758c165b41a6ededfe9b80bc8a919dd76ac6b692c47d89c5d1ecc888ad794"}
May 26 17:04:47.018753 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:47.018734    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:47.830153 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:47.829969    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:47.830307 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:47.830226    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:48.009083 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.009052    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:04:48.009660 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.009432    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" event={"ID":"2753c993-0102-4e57-b557-f03934174f89","Type":"ContainerStarted","Data":"8cf4f74e97538e245e4bdc0bdc61a369d34e239f4ab3f22744b61d2e784e6ead"}
May 26 17:04:48.009727 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.009684    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:48.011540 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.011518    2571 generic.go:358] "Generic (PLEG): container finished" podID="a60644b9-b57b-4023-bc10-ddb686016835" containerID="2c1940791631786cee5bb1a11501453fd35145a514ddea3cfe790f2d1a626085" exitCode=0
May 26 17:04:48.011652 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.011570    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" event={"ID":"a60644b9-b57b-4023-bc10-ddb686016835","Type":"ContainerDied","Data":"2c1940791631786cee5bb1a11501453fd35145a514ddea3cfe790f2d1a626085"}
May 26 17:04:48.024973 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.024953    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:04:48.038475 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.038433    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2" podStartSLOduration=10.340157439 podStartE2EDuration="29.038419178s" podCreationTimestamp="2026-05-26 17:04:19 +0000 UTC" firstStartedPulling="2026-05-26 17:04:22.425434878 +0000 UTC m=+3.150524357" lastFinishedPulling="2026-05-26 17:04:41.123696598 +0000 UTC m=+21.848786096" observedRunningTime="2026-05-26 17:04:48.03700598 +0000 UTC m=+28.762095483" watchObservedRunningTime="2026-05-26 17:04:48.038419178 +0000 UTC m=+28.763508704"
May 26 17:04:48.168063 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.167993    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fjgrn"]
May 26 17:04:48.168226 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.168147    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:48.168290 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:48.168243    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:48.168698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.168676    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lztjw"]
May 26 17:04:48.168793 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:48.168781    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:48.168885 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:48.168867    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:49.015576 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:49.015547    2571 generic.go:358] "Generic (PLEG): container finished" podID="a60644b9-b57b-4023-bc10-ddb686016835" containerID="8e2654c3d8b6b14989dd146bc498d8bd1eba54bd3e13e0bcd42223c1459011e9" exitCode=0
May 26 17:04:49.016051 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:49.015634    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" event={"ID":"a60644b9-b57b-4023-bc10-ddb686016835","Type":"ContainerDied","Data":"8e2654c3d8b6b14989dd146bc498d8bd1eba54bd3e13e0bcd42223c1459011e9"}
May 26 17:04:49.832570 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:49.831452    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:49.832570 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:49.831589    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:49.832570 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:49.832060    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:49.832570 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:49.832204    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:51.829970 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:51.829936    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:51.830549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:51.829937    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:51.830549 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:51.830087    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjgrn" podUID="9bd326f5-4fac-48da-b113-83b2234bc506"
May 26 17:04:51.830549 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:51.830141    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:04:53.472731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.472683    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:53.473215 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.472859    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:53.473215 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.472943    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs podName:a13dbe45-e72d-4315-a2a1-c4736202fd9b nodeName:}" failed. No retries permitted until 2026-05-26 17:05:25.472922753 +0000 UTC m=+66.198012246 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs") pod "network-metrics-daemon-lztjw" (UID: "a13dbe45-e72d-4315-a2a1-c4736202fd9b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 17:04:53.643028 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.642935    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-109.ec2.internal" event="NodeReady"
May 26 17:04:53.643225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.643131    2571 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 17:04:53.674479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.674443    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:53.674687 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.674628    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 17:04:53.674687 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.674648    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 17:04:53.674687 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.674659    2571 projected.go:194] Error preparing data for projected volume kube-api-access-657zs for pod openshift-network-diagnostics/network-check-target-fjgrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:53.674831 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.674718    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs podName:9bd326f5-4fac-48da-b113-83b2234bc506 nodeName:}" failed. No retries permitted until 2026-05-26 17:05:25.674699485 +0000 UTC m=+66.399788982 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-657zs" (UniqueName: "kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs") pod "network-check-target-fjgrn" (UID: "9bd326f5-4fac-48da-b113-83b2234bc506") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 17:04:53.689125 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.689085    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4c8fb"]
May 26 17:04:53.714524 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.714455    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mvfcs"]
May 26 17:04:53.714712 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.714649    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.717438 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.717415    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 17:04:53.717741 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.717720    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 17:04:53.717741 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.717737    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pnxz5\""
May 26 17:04:53.728467 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.728444    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4c8fb"]
May 26 17:04:53.728588 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.728474    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mvfcs"]
May 26 17:04:53.728588 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.728576    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:04:53.731479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.731323    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 17:04:53.731479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.731380    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 17:04:53.731479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.731399    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 17:04:53.731479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.731422    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jvpqs\""
May 26 17:04:53.830443 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.830405    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:04:53.830724 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.830697    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:04:53.833698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.833673    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pvds2\""
May 26 17:04:53.833959 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.833938    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 17:04:53.834098 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.833972    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brj54\""
May 26 17:04:53.834098 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.834001    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 17:04:53.834282 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.834262    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 17:04:53.875304 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.875271    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:04:53.875479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.875335    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-config-volume\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.875479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.875368    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-tmp-dir\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.875479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.875409    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.875479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.875430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z687m\" (UniqueName: \"kubernetes.io/projected/b5935fea-b672-471d-a707-a16fc6f3203e-kube-api-access-z687m\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:04:53.875691 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.875482    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgp8\" (UniqueName: \"kubernetes.io/projected/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-kube-api-access-qzgp8\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.976453 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.976373    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgp8\" (UniqueName: \"kubernetes.io/projected/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-kube-api-access-qzgp8\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.976661 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.976557    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:04:53.976661 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.976598    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-config-volume\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.976661 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.976626    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-tmp-dir\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.976844 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.976673    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.976844 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.976703    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z687m\" (UniqueName: \"kubernetes.io/projected/b5935fea-b672-471d-a707-a16fc6f3203e-kube-api-access-z687m\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:04:53.976844 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.976731    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 17:04:53.976844 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.976805    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert podName:b5935fea-b672-471d-a707-a16fc6f3203e nodeName:}" failed. No retries permitted until 2026-05-26 17:04:54.476784226 +0000 UTC m=+35.201873725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert") pod "ingress-canary-mvfcs" (UID: "b5935fea-b672-471d-a707-a16fc6f3203e") : secret "canary-serving-cert" not found
May 26 17:04:53.977097 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.976991    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 17:04:53.977097 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:53.977051    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls podName:91c3c9fd-a70c-4e23-85db-d5544f5c9a73 nodeName:}" failed. No retries permitted until 2026-05-26 17:04:54.477035825 +0000 UTC m=+35.202125328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls") pod "dns-default-4c8fb" (UID: "91c3c9fd-a70c-4e23-85db-d5544f5c9a73") : secret "dns-default-metrics-tls" not found
May 26 17:04:53.977214 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.977130    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-tmp-dir\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.977574 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.977539    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-config-volume\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.986998 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.986974    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgp8\" (UniqueName: \"kubernetes.io/projected/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-kube-api-access-qzgp8\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:53.987229 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:53.987209    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z687m\" (UniqueName: \"kubernetes.io/projected/b5935fea-b672-471d-a707-a16fc6f3203e-kube-api-access-z687m\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:04:54.480643 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:54.480601    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:54.481230 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:54.480702    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:04:54.481230 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:54.480779    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 17:04:54.481230 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:54.480816    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 17:04:54.481230 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:54.480859    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls podName:91c3c9fd-a70c-4e23-85db-d5544f5c9a73 nodeName:}" failed. No retries permitted until 2026-05-26 17:04:55.480837299 +0000 UTC m=+36.205926801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls") pod "dns-default-4c8fb" (UID: "91c3c9fd-a70c-4e23-85db-d5544f5c9a73") : secret "dns-default-metrics-tls" not found
May 26 17:04:54.481230 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:54.480874    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert podName:b5935fea-b672-471d-a707-a16fc6f3203e nodeName:}" failed. No retries permitted until 2026-05-26 17:04:55.48086806 +0000 UTC m=+36.205957539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert") pod "ingress-canary-mvfcs" (UID: "b5935fea-b672-471d-a707-a16fc6f3203e") : secret "canary-serving-cert" not found
May 26 17:04:55.489143 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:55.489104    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:04:55.489539 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:55.489163    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:55.489539 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:55.489248    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 17:04:55.489539 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:55.489319    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert podName:b5935fea-b672-471d-a707-a16fc6f3203e nodeName:}" failed. No retries permitted until 2026-05-26 17:04:57.489303051 +0000 UTC m=+38.214392546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert") pod "ingress-canary-mvfcs" (UID: "b5935fea-b672-471d-a707-a16fc6f3203e") : secret "canary-serving-cert" not found
May 26 17:04:55.489539 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:55.489260    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 17:04:55.489539 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:55.489375    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls podName:91c3c9fd-a70c-4e23-85db-d5544f5c9a73 nodeName:}" failed. No retries permitted until 2026-05-26 17:04:57.489360829 +0000 UTC m=+38.214450308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls") pod "dns-default-4c8fb" (UID: "91c3c9fd-a70c-4e23-85db-d5544f5c9a73") : secret "dns-default-metrics-tls" not found
May 26 17:04:56.033265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:56.033234    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" event={"ID":"a60644b9-b57b-4023-bc10-ddb686016835","Type":"ContainerStarted","Data":"0e76d1a55d9e3a396a00ad7cfad3178ebf36aeccdf15ce7cbac4d675a22243f9"}
May 26 17:04:57.037137 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:57.037103    2571 generic.go:358] "Generic (PLEG): container finished" podID="a60644b9-b57b-4023-bc10-ddb686016835" containerID="0e76d1a55d9e3a396a00ad7cfad3178ebf36aeccdf15ce7cbac4d675a22243f9" exitCode=0
May 26 17:04:57.037567 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:57.037152    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" event={"ID":"a60644b9-b57b-4023-bc10-ddb686016835","Type":"ContainerDied","Data":"0e76d1a55d9e3a396a00ad7cfad3178ebf36aeccdf15ce7cbac4d675a22243f9"}
May 26 17:04:57.504875 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:57.504829    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:04:57.505067 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:57.504911    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:04:57.505067 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:57.504991    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 17:04:57.505067 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:57.505064    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert podName:b5935fea-b672-471d-a707-a16fc6f3203e nodeName:}" failed. No retries permitted until 2026-05-26 17:05:01.505050144 +0000 UTC m=+42.230139626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert") pod "ingress-canary-mvfcs" (UID: "b5935fea-b672-471d-a707-a16fc6f3203e") : secret "canary-serving-cert" not found
May 26 17:04:57.505185 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:57.504991    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 17:04:57.505185 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:04:57.505154    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls podName:91c3c9fd-a70c-4e23-85db-d5544f5c9a73 nodeName:}" failed. No retries permitted until 2026-05-26 17:05:01.50513636 +0000 UTC m=+42.230225855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls") pod "dns-default-4c8fb" (UID: "91c3c9fd-a70c-4e23-85db-d5544f5c9a73") : secret "dns-default-metrics-tls" not found
May 26 17:04:58.041049 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:58.040998    2571 generic.go:358] "Generic (PLEG): container finished" podID="a60644b9-b57b-4023-bc10-ddb686016835" containerID="631db514276f4875cac7b22f68389c49ccfe9e43487413cf1fc842c38300fd85" exitCode=0
May 26 17:04:58.041433 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:58.041050    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" event={"ID":"a60644b9-b57b-4023-bc10-ddb686016835","Type":"ContainerDied","Data":"631db514276f4875cac7b22f68389c49ccfe9e43487413cf1fc842c38300fd85"}
May 26 17:04:59.046221 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:59.046182    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" event={"ID":"a60644b9-b57b-4023-bc10-ddb686016835","Type":"ContainerStarted","Data":"90d321e4cb6207b5095899ef9d645661eef29c92f32c06f566502c4f1c628203"}
May 26 17:04:59.070897 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:04:59.069119    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8qh7j" podStartSLOduration=5.598233184 podStartE2EDuration="39.069100952s" podCreationTimestamp="2026-05-26 17:04:20 +0000 UTC" firstStartedPulling="2026-05-26 17:04:22.421663826 +0000 UTC m=+3.146753308" lastFinishedPulling="2026-05-26 17:04:55.892531582 +0000 UTC m=+36.617621076" observedRunningTime="2026-05-26 17:04:59.068598932 +0000 UTC m=+39.793688446" watchObservedRunningTime="2026-05-26 17:04:59.069100952 +0000 UTC m=+39.794190455"
May 26 17:05:01.536805 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:01.536761    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:05:01.537214 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:01.536843    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:05:01.537214 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:01.536921    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 17:05:01.537214 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:01.536931    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 17:05:01.537214 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:01.536990    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls podName:91c3c9fd-a70c-4e23-85db-d5544f5c9a73 nodeName:}" failed. No retries permitted until 2026-05-26 17:05:09.536973882 +0000 UTC m=+50.262063378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls") pod "dns-default-4c8fb" (UID: "91c3c9fd-a70c-4e23-85db-d5544f5c9a73") : secret "dns-default-metrics-tls" not found
May 26 17:05:01.537214 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:01.537005    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert podName:b5935fea-b672-471d-a707-a16fc6f3203e nodeName:}" failed. No retries permitted until 2026-05-26 17:05:09.536999415 +0000 UTC m=+50.262088905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert") pod "ingress-canary-mvfcs" (UID: "b5935fea-b672-471d-a707-a16fc6f3203e") : secret "canary-serving-cert" not found
May 26 17:05:09.589325 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:09.589276    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:05:09.589712 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:09.589365    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:05:09.589712 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:09.589440    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 17:05:09.589712 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:09.589518    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert podName:b5935fea-b672-471d-a707-a16fc6f3203e nodeName:}" failed. No retries permitted until 2026-05-26 17:05:25.589500099 +0000 UTC m=+66.314589590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert") pod "ingress-canary-mvfcs" (UID: "b5935fea-b672-471d-a707-a16fc6f3203e") : secret "canary-serving-cert" not found
May 26 17:05:09.589712 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:09.589440    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 17:05:09.589712 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:09.589546    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls podName:91c3c9fd-a70c-4e23-85db-d5544f5c9a73 nodeName:}" failed. No retries permitted until 2026-05-26 17:05:25.589539869 +0000 UTC m=+66.314629361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls") pod "dns-default-4c8fb" (UID: "91c3c9fd-a70c-4e23-85db-d5544f5c9a73") : secret "dns-default-metrics-tls" not found
May 26 17:05:20.027787 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:20.027753    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r56x2"
May 26 17:05:25.497686 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.497642    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:05:25.500844 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.500821    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 17:05:25.508244 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:25.508215    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 17:05:25.508338 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:25.508300    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs podName:a13dbe45-e72d-4315-a2a1-c4736202fd9b nodeName:}" failed. No retries permitted until 2026-05-26 17:06:29.508276281 +0000 UTC m=+130.233365775 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs") pod "network-metrics-daemon-lztjw" (UID: "a13dbe45-e72d-4315-a2a1-c4736202fd9b") : secret "metrics-daemon-secret" not found
May 26 17:05:25.598981 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.598938    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:05:25.599196 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.598997    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:05:25.599196 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:25.599114    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 17:05:25.599315 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:25.599114    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 17:05:25.599315 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:25.599204    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls podName:91c3c9fd-a70c-4e23-85db-d5544f5c9a73 nodeName:}" failed. No retries permitted until 2026-05-26 17:05:57.59918248 +0000 UTC m=+98.324271975 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls") pod "dns-default-4c8fb" (UID: "91c3c9fd-a70c-4e23-85db-d5544f5c9a73") : secret "dns-default-metrics-tls" not found
May 26 17:05:25.599315 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:25.599267    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert podName:b5935fea-b672-471d-a707-a16fc6f3203e nodeName:}" failed. No retries permitted until 2026-05-26 17:05:57.599252634 +0000 UTC m=+98.324342112 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert") pod "ingress-canary-mvfcs" (UID: "b5935fea-b672-471d-a707-a16fc6f3203e") : secret "canary-serving-cert" not found
May 26 17:05:25.700352 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.700307    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:05:25.703390 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.703368    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 17:05:25.713120 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.713098    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 17:05:25.724920 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.724894    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657zs\" (UniqueName: \"kubernetes.io/projected/9bd326f5-4fac-48da-b113-83b2234bc506-kube-api-access-657zs\") pod \"network-check-target-fjgrn\" (UID: \"9bd326f5-4fac-48da-b113-83b2234bc506\") " pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:05:25.946590 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.946495    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pvds2\""
May 26 17:05:25.953944 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:25.953921    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:05:26.085150 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:26.085115    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fjgrn"]
May 26 17:05:26.089327 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:05:26.089286    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd326f5_4fac_48da_b113_83b2234bc506.slice/crio-38bec7b57e365275d86e59d85270a7b881d7622ef413491af2f83e2831b7fbd2 WatchSource:0}: Error finding container 38bec7b57e365275d86e59d85270a7b881d7622ef413491af2f83e2831b7fbd2: Status 404 returned error can't find the container with id 38bec7b57e365275d86e59d85270a7b881d7622ef413491af2f83e2831b7fbd2
May 26 17:05:26.096581 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:26.096551    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fjgrn" event={"ID":"9bd326f5-4fac-48da-b113-83b2234bc506","Type":"ContainerStarted","Data":"38bec7b57e365275d86e59d85270a7b881d7622ef413491af2f83e2831b7fbd2"}
May 26 17:05:29.104591 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:29.104550    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fjgrn" event={"ID":"9bd326f5-4fac-48da-b113-83b2234bc506","Type":"ContainerStarted","Data":"098ebf9ae4fabe8e49947c0f4c1d2bd99e57c3c9d87edef30c40a36a0ea60f50"}
May 26 17:05:29.104971 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:29.104667    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:05:29.120695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:29.120653    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fjgrn" podStartSLOduration=66.256059578 podStartE2EDuration="1m9.120638205s" podCreationTimestamp="2026-05-26 17:04:20 +0000 UTC" firstStartedPulling="2026-05-26 17:05:26.091345189 +0000 UTC m=+66.816434683" lastFinishedPulling="2026-05-26 17:05:28.955923827 +0000 UTC m=+69.681013310" observedRunningTime="2026-05-26 17:05:29.119471883 +0000 UTC m=+69.844561383" watchObservedRunningTime="2026-05-26 17:05:29.120638205 +0000 UTC m=+69.845727749"
May 26 17:05:57.612734 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:57.612585    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:05:57.612734 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:05:57.612659    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:05:57.613284 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:57.612740    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 17:05:57.613284 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:57.612775    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 17:05:57.613284 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:57.612825    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert podName:b5935fea-b672-471d-a707-a16fc6f3203e nodeName:}" failed. No retries permitted until 2026-05-26 17:07:01.612809109 +0000 UTC m=+162.337898603 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert") pod "ingress-canary-mvfcs" (UID: "b5935fea-b672-471d-a707-a16fc6f3203e") : secret "canary-serving-cert" not found
May 26 17:05:57.613284 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:05:57.612839    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls podName:91c3c9fd-a70c-4e23-85db-d5544f5c9a73 nodeName:}" failed. No retries permitted until 2026-05-26 17:07:01.612832935 +0000 UTC m=+162.337922414 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls") pod "dns-default-4c8fb" (UID: "91c3c9fd-a70c-4e23-85db-d5544f5c9a73") : secret "dns-default-metrics-tls" not found
May 26 17:06:00.108881 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:00.108851    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fjgrn"
May 26 17:06:08.269713 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.269673    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"]
May 26 17:06:08.271695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.271675    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.277122 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.277086    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 26 17:06:08.277122 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.277109    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 26 17:06:08.277317 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.277126    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 17:06:08.277317 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.277090    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 17:06:08.277317 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.277088    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tjfkb\""
May 26 17:06:08.283334 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.283311    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"]
May 26 17:06:08.369326 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.369291    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb"]
May 26 17:06:08.371215 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.371198    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb"
May 26 17:06:08.373993 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.373970    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fnrkw\""
May 26 17:06:08.374147 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.373987    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 26 17:06:08.374648 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.374634    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 26 17:06:08.376435 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.376273    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-w47qs"]
May 26 17:06:08.378152 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.378132    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-978fdbd8-97ct5"]
May 26 17:06:08.378320 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.378299    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.379915 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.379897    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb"]
May 26 17:06:08.380036 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.380006    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.382375 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.382357    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 26 17:06:08.382854 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.382836    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6k4zj\""
May 26 17:06:08.382925 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.382843    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 17:06:08.383130 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.383115    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 26 17:06:08.383313 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.383295    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 17:06:08.383426 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.383320    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 26 17:06:08.383426 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.383351    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 26 17:06:08.383426 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.383352    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 26 17:06:08.383426 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.383387    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 26 17:06:08.383638 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.383588    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 26 17:06:08.383830 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.383814    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 26 17:06:08.384215 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.384199    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-96mvf\""
May 26 17:06:08.385290 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.385250    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.385502 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.385322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/38027300-8fba-4a1d-8f9c-745e9942954a-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.385502 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.385351    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254nn\" (UniqueName: \"kubernetes.io/projected/38027300-8fba-4a1d-8f9c-745e9942954a-kube-api-access-254nn\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.388598 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.388578    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 26 17:06:08.394675 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.394564    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-w47qs"]
May 26 17:06:08.395320 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.395299    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-978fdbd8-97ct5"]
May 26 17:06:08.470965 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.470931    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"]
May 26 17:06:08.473087 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.473071    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:08.475608 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.475587    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 26 17:06:08.475856 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.475843    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 26 17:06:08.475912 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.475898    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 26 17:06:08.475963 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.475947    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6wdf9\""
May 26 17:06:08.484716 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.484684    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"]
May 26 17:06:08.485853 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.485830    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fznt8\" (UniqueName: \"kubernetes.io/projected/1f1f5c29-30bc-48b5-96b8-2d130fa358c1-kube-api-access-fznt8\") pod \"volume-data-source-validator-5cf8c89d64-pg9gb\" (UID: \"1f1f5c29-30bc-48b5-96b8-2d130fa358c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb"
May 26 17:06:08.485949 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.485860    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f96e4a7-3baa-47a8-b688-ed682aaeba85-tmp\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.485949 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.485878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f96e4a7-3baa-47a8-b688-ed682aaeba85-service-ca-bundle\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.485949 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.485897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-stats-auth\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.486092 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.485974    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f96e4a7-3baa-47a8-b688-ed682aaeba85-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.486092 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486005    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.486092 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486070    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f96e4a7-3baa-47a8-b688-ed682aaeba85-snapshots\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.486092 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486091    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.486233 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486109    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqh68\" (UniqueName: \"kubernetes.io/projected/5f96e4a7-3baa-47a8-b688-ed682aaeba85-kube-api-access-rqh68\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.486233 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486128    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-default-certificate\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.486233 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486144    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7td\" (UniqueName: \"kubernetes.io/projected/9008dab9-05e5-43b6-bdc8-077476da5900-kube-api-access-5h7td\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.486233 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486208    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.486357 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f96e4a7-3baa-47a8-b688-ed682aaeba85-serving-cert\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.486357 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:08.486269    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 17:06:08.486357 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486285    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/38027300-8fba-4a1d-8f9c-745e9942954a-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.486357 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486301    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-254nn\" (UniqueName: \"kubernetes.io/projected/38027300-8fba-4a1d-8f9c-745e9942954a-kube-api-access-254nn\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.486357 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:08.486338    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls podName:38027300-8fba-4a1d-8f9c-745e9942954a nodeName:}" failed. No retries permitted until 2026-05-26 17:06:08.986318549 +0000 UTC m=+109.711408049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-99qnk" (UID: "38027300-8fba-4a1d-8f9c-745e9942954a") : secret "cluster-monitoring-operator-tls" not found
May 26 17:06:08.486940 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.486922    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/38027300-8fba-4a1d-8f9c-745e9942954a-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.498749 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.498720    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254nn\" (UniqueName: \"kubernetes.io/projected/38027300-8fba-4a1d-8f9c-745e9942954a-kube-api-access-254nn\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.587334 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587225    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-stats-auth\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.587334 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587283    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f96e4a7-3baa-47a8-b688-ed682aaeba85-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.587549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587405    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.587549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587447    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f96e4a7-3baa-47a8-b688-ed682aaeba85-snapshots\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.587549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587466    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.587549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587485    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rqh68\" (UniqueName: \"kubernetes.io/projected/5f96e4a7-3baa-47a8-b688-ed682aaeba85-kube-api-access-rqh68\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.587549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587503    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-default-certificate\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.587549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587523    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7td\" (UniqueName: \"kubernetes.io/projected/9008dab9-05e5-43b6-bdc8-077476da5900-kube-api-access-5h7td\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.587902 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:08.587545    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 17:06:08.587902 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587555    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:08.587902 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:08.587627    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:09.087606331 +0000 UTC m=+109.812695826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : secret "router-metrics-certs-default" not found
May 26 17:06:08.587902 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587653    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82sgn\" (UniqueName: \"kubernetes.io/projected/937078be-09eb-4f20-9ef7-eab92c00b8bd-kube-api-access-82sgn\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:08.587902 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:08.587661    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:09.087640705 +0000 UTC m=+109.812730201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : configmap references non-existent config key: service-ca.crt
May 26 17:06:08.587902 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587748    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f96e4a7-3baa-47a8-b688-ed682aaeba85-serving-cert\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.587902 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587811    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fznt8\" (UniqueName: \"kubernetes.io/projected/1f1f5c29-30bc-48b5-96b8-2d130fa358c1-kube-api-access-fznt8\") pod \"volume-data-source-validator-5cf8c89d64-pg9gb\" (UID: \"1f1f5c29-30bc-48b5-96b8-2d130fa358c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb"
May 26 17:06:08.587902 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587861    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f96e4a7-3baa-47a8-b688-ed682aaeba85-tmp\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.587902 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.587896    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f96e4a7-3baa-47a8-b688-ed682aaeba85-service-ca-bundle\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.588368 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.588146    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f96e4a7-3baa-47a8-b688-ed682aaeba85-snapshots\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.588368 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.588253    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f96e4a7-3baa-47a8-b688-ed682aaeba85-tmp\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.588544 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.588523    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f96e4a7-3baa-47a8-b688-ed682aaeba85-service-ca-bundle\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.588878 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.588858    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f96e4a7-3baa-47a8-b688-ed682aaeba85-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.589824 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.589802    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-stats-auth\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.590089 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.590071    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f96e4a7-3baa-47a8-b688-ed682aaeba85-serving-cert\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.590607 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.590589    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-default-certificate\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.596678 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.596656    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqh68\" (UniqueName: \"kubernetes.io/projected/5f96e4a7-3baa-47a8-b688-ed682aaeba85-kube-api-access-rqh68\") pod \"insights-operator-78bdfb888c-w47qs\" (UID: \"5f96e4a7-3baa-47a8-b688-ed682aaeba85\") " pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.597335 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.597315    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7td\" (UniqueName: \"kubernetes.io/projected/9008dab9-05e5-43b6-bdc8-077476da5900-kube-api-access-5h7td\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:08.597412 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.597372    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fznt8\" (UniqueName: \"kubernetes.io/projected/1f1f5c29-30bc-48b5-96b8-2d130fa358c1-kube-api-access-fznt8\") pod \"volume-data-source-validator-5cf8c89d64-pg9gb\" (UID: \"1f1f5c29-30bc-48b5-96b8-2d130fa358c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb"
May 26 17:06:08.681753 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.681705    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb"
May 26 17:06:08.688674 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.688647    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:08.688786 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.688701    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82sgn\" (UniqueName: \"kubernetes.io/projected/937078be-09eb-4f20-9ef7-eab92c00b8bd-kube-api-access-82sgn\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:08.688830 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:08.688803    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 17:06:08.688886 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:08.688874    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls podName:937078be-09eb-4f20-9ef7-eab92c00b8bd nodeName:}" failed. No retries permitted until 2026-05-26 17:06:09.188855899 +0000 UTC m=+109.913945393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls") pod "cluster-samples-operator-697587599-cvqdr" (UID: "937078be-09eb-4f20-9ef7-eab92c00b8bd") : secret "samples-operator-tls" not found
May 26 17:06:08.690597 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.690572    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-w47qs"
May 26 17:06:08.697869 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.697847    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82sgn\" (UniqueName: \"kubernetes.io/projected/937078be-09eb-4f20-9ef7-eab92c00b8bd-kube-api-access-82sgn\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:08.828527 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.828488    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb"]
May 26 17:06:08.831480 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:08.831450    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1f5c29_30bc_48b5_96b8_2d130fa358c1.slice/crio-9395e8d813774e9d612f43623883d9cfffc4d20e0f9971b44eb11d3f2dd398b0 WatchSource:0}: Error finding container 9395e8d813774e9d612f43623883d9cfffc4d20e0f9971b44eb11d3f2dd398b0: Status 404 returned error can't find the container with id 9395e8d813774e9d612f43623883d9cfffc4d20e0f9971b44eb11d3f2dd398b0
May 26 17:06:08.844170 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.844110    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-w47qs"]
May 26 17:06:08.848303 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:08.848279    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f96e4a7_3baa_47a8_b688_ed682aaeba85.slice/crio-0b4138bc349c5abddecfdfce2fbca5653de5b46acc4d7f56d91af9a88d719915 WatchSource:0}: Error finding container 0b4138bc349c5abddecfdfce2fbca5653de5b46acc4d7f56d91af9a88d719915: Status 404 returned error can't find the container with id 0b4138bc349c5abddecfdfce2fbca5653de5b46acc4d7f56d91af9a88d719915
May 26 17:06:08.992056 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:08.991993    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:08.992223 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:08.992143    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 17:06:08.992223 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:08.992214    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls podName:38027300-8fba-4a1d-8f9c-745e9942954a nodeName:}" failed. No retries permitted until 2026-05-26 17:06:09.992189985 +0000 UTC m=+110.717279465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-99qnk" (UID: "38027300-8fba-4a1d-8f9c-745e9942954a") : secret "cluster-monitoring-operator-tls" not found
May 26 17:06:09.092849 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:09.092814    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:09.092849 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:09.092855    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:09.093132 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:09.093001    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 17:06:09.093132 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:09.093104    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:10.093080783 +0000 UTC m=+110.818170298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : secret "router-metrics-certs-default" not found
May 26 17:06:09.093132 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:09.093124    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:10.093114587 +0000 UTC m=+110.818204096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : configmap references non-existent config key: service-ca.crt
May 26 17:06:09.184214 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:09.184124    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-w47qs" event={"ID":"5f96e4a7-3baa-47a8-b688-ed682aaeba85","Type":"ContainerStarted","Data":"0b4138bc349c5abddecfdfce2fbca5653de5b46acc4d7f56d91af9a88d719915"}
May 26 17:06:09.185041 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:09.185003    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb" event={"ID":"1f1f5c29-30bc-48b5-96b8-2d130fa358c1","Type":"ContainerStarted","Data":"9395e8d813774e9d612f43623883d9cfffc4d20e0f9971b44eb11d3f2dd398b0"}
May 26 17:06:09.194376 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:09.194333    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:09.194493 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:09.194482    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 17:06:09.194547 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:09.194537    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls podName:937078be-09eb-4f20-9ef7-eab92c00b8bd nodeName:}" failed. No retries permitted until 2026-05-26 17:06:10.194521442 +0000 UTC m=+110.919610930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls") pod "cluster-samples-operator-697587599-cvqdr" (UID: "937078be-09eb-4f20-9ef7-eab92c00b8bd") : secret "samples-operator-tls" not found
May 26 17:06:10.002561 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:10.002519    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:10.003001 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:10.002648    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 17:06:10.003001 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:10.002724    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls podName:38027300-8fba-4a1d-8f9c-745e9942954a nodeName:}" failed. No retries permitted until 2026-05-26 17:06:12.002704907 +0000 UTC m=+112.727794399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-99qnk" (UID: "38027300-8fba-4a1d-8f9c-745e9942954a") : secret "cluster-monitoring-operator-tls" not found
May 26 17:06:10.103700 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:10.103664    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:10.103871 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:10.103718    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:10.103871 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:10.103844    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 17:06:10.103989 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:10.103922    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:12.103898971 +0000 UTC m=+112.828988449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : secret "router-metrics-certs-default" not found
May 26 17:06:10.104055 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:10.103999    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:12.103980754 +0000 UTC m=+112.829070247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : configmap references non-existent config key: service-ca.crt
May 26 17:06:10.204574 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:10.204531    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:10.204773 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:10.204699    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 17:06:10.204856 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:10.204779    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls podName:937078be-09eb-4f20-9ef7-eab92c00b8bd nodeName:}" failed. No retries permitted until 2026-05-26 17:06:12.204756424 +0000 UTC m=+112.929845904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls") pod "cluster-samples-operator-697587599-cvqdr" (UID: "937078be-09eb-4f20-9ef7-eab92c00b8bd") : secret "samples-operator-tls" not found
May 26 17:06:11.190569 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:11.190529    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb" event={"ID":"1f1f5c29-30bc-48b5-96b8-2d130fa358c1","Type":"ContainerStarted","Data":"273416da23d9e0789b3384e158b4c298338406a863e69405577a7ab9732f8942"}
May 26 17:06:11.191881 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:11.191857    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-w47qs" event={"ID":"5f96e4a7-3baa-47a8-b688-ed682aaeba85","Type":"ContainerStarted","Data":"341919872b75a990d34f85e111bc4be7a7f05687aa099603851ffe866177fdf7"}
May 26 17:06:11.206810 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:11.206767    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-pg9gb" podStartSLOduration=1.748215086 podStartE2EDuration="3.206752903s" podCreationTimestamp="2026-05-26 17:06:08 +0000 UTC" firstStartedPulling="2026-05-26 17:06:08.833209731 +0000 UTC m=+109.558299210" lastFinishedPulling="2026-05-26 17:06:10.291747544 +0000 UTC m=+111.016837027" observedRunningTime="2026-05-26 17:06:11.206217339 +0000 UTC m=+111.931306840" watchObservedRunningTime="2026-05-26 17:06:11.206752903 +0000 UTC m=+111.931842405"
May 26 17:06:11.228668 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:11.226003    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-w47qs" podStartSLOduration=1.07741042 podStartE2EDuration="3.22596071s" podCreationTimestamp="2026-05-26 17:06:08 +0000 UTC" firstStartedPulling="2026-05-26 17:06:08.849907283 +0000 UTC m=+109.574996762" lastFinishedPulling="2026-05-26 17:06:10.998457573 +0000 UTC m=+111.723547052" observedRunningTime="2026-05-26 17:06:11.224513486 +0000 UTC m=+111.949602987" watchObservedRunningTime="2026-05-26 17:06:11.22596071 +0000 UTC m=+111.951050212"
May 26 17:06:12.018996 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:12.018957    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:12.019199 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:12.019115    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 17:06:12.019199 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:12.019186    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls podName:38027300-8fba-4a1d-8f9c-745e9942954a nodeName:}" failed. No retries permitted until 2026-05-26 17:06:16.019170167 +0000 UTC m=+116.744259649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-99qnk" (UID: "38027300-8fba-4a1d-8f9c-745e9942954a") : secret "cluster-monitoring-operator-tls" not found
May 26 17:06:12.120138 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:12.120103    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:12.120138 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:12.120146    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:12.120391 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:12.120253    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:16.120234182 +0000 UTC m=+116.845323675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : configmap references non-existent config key: service-ca.crt
May 26 17:06:12.120391 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:12.120256    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 17:06:12.120391 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:12.120319    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:16.120299508 +0000 UTC m=+116.845389002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : secret "router-metrics-certs-default" not found
May 26 17:06:12.220905 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:12.220872    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:12.221393 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:12.221058    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 17:06:12.221393 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:12.221139    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls podName:937078be-09eb-4f20-9ef7-eab92c00b8bd nodeName:}" failed. No retries permitted until 2026-05-26 17:06:16.221119614 +0000 UTC m=+116.946209110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls") pod "cluster-samples-operator-697587599-cvqdr" (UID: "937078be-09eb-4f20-9ef7-eab92c00b8bd") : secret "samples-operator-tls" not found
May 26 17:06:13.270886 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.270851    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"]
May 26 17:06:13.272836 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.272821    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.275458 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.275432    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 26 17:06:13.275458 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.275445    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 26 17:06:13.275646 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.275440    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-twkcp\""
May 26 17:06:13.275646 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.275429    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 26 17:06:13.276581 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.276566    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 26 17:06:13.288473 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.288444    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"]
May 26 17:06:13.329370 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.329336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df3cb020-a879-45c7-8fff-b38a4b5ccf30-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-gbm7m\" (UID: \"df3cb020-a879-45c7-8fff-b38a4b5ccf30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.329573 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.329398    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df3cb020-a879-45c7-8fff-b38a4b5ccf30-config\") pod \"kube-storage-version-migrator-operator-565644f49-gbm7m\" (UID: \"df3cb020-a879-45c7-8fff-b38a4b5ccf30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.329573 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.329465    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f466z\" (UniqueName: \"kubernetes.io/projected/df3cb020-a879-45c7-8fff-b38a4b5ccf30-kube-api-access-f466z\") pod \"kube-storage-version-migrator-operator-565644f49-gbm7m\" (UID: \"df3cb020-a879-45c7-8fff-b38a4b5ccf30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.429849 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.429807    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f466z\" (UniqueName: \"kubernetes.io/projected/df3cb020-a879-45c7-8fff-b38a4b5ccf30-kube-api-access-f466z\") pod \"kube-storage-version-migrator-operator-565644f49-gbm7m\" (UID: \"df3cb020-a879-45c7-8fff-b38a4b5ccf30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.430080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.429924    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df3cb020-a879-45c7-8fff-b38a4b5ccf30-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-gbm7m\" (UID: \"df3cb020-a879-45c7-8fff-b38a4b5ccf30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.430080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.429951    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df3cb020-a879-45c7-8fff-b38a4b5ccf30-config\") pod \"kube-storage-version-migrator-operator-565644f49-gbm7m\" (UID: \"df3cb020-a879-45c7-8fff-b38a4b5ccf30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.430456 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.430439    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df3cb020-a879-45c7-8fff-b38a4b5ccf30-config\") pod \"kube-storage-version-migrator-operator-565644f49-gbm7m\" (UID: \"df3cb020-a879-45c7-8fff-b38a4b5ccf30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.432148 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.432132    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df3cb020-a879-45c7-8fff-b38a4b5ccf30-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-gbm7m\" (UID: \"df3cb020-a879-45c7-8fff-b38a4b5ccf30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.438124 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.438100    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f466z\" (UniqueName: \"kubernetes.io/projected/df3cb020-a879-45c7-8fff-b38a4b5ccf30-kube-api-access-f466z\") pod \"kube-storage-version-migrator-operator-565644f49-gbm7m\" (UID: \"df3cb020-a879-45c7-8fff-b38a4b5ccf30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.582449 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.582359    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"
May 26 17:06:13.703714 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.703679    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m"]
May 26 17:06:13.706786 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:13.706751    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3cb020_a879_45c7_8fff_b38a4b5ccf30.slice/crio-51b2ad4df8e763a87878d96f965f273d2248119fd3272e1314a7cbbbddc8c677 WatchSource:0}: Error finding container 51b2ad4df8e763a87878d96f965f273d2248119fd3272e1314a7cbbbddc8c677: Status 404 returned error can't find the container with id 51b2ad4df8e763a87878d96f965f273d2248119fd3272e1314a7cbbbddc8c677
May 26 17:06:13.997008 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:13.996982    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m97tn_87203750-2eb1-4b8c-9c94-b7ca88363b40/dns-node-resolver/0.log"
May 26 17:06:14.198420 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:14.198386    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m" event={"ID":"df3cb020-a879-45c7-8fff-b38a4b5ccf30","Type":"ContainerStarted","Data":"51b2ad4df8e763a87878d96f965f273d2248119fd3272e1314a7cbbbddc8c677"}
May 26 17:06:14.596266 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:14.596240    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6w2fz_f9c75f58-61b8-4762-a4b4-ceef1c52bf7b/node-ca/0.log"
May 26 17:06:15.255686 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.255641    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"]
May 26 17:06:15.257666 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.257642    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.260432 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.260409    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 26 17:06:15.260432 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.260427    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 26 17:06:15.260630 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.260434    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 26 17:06:15.261584 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.261562    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 26 17:06:15.261687 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.261563    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hjlc5\""
May 26 17:06:15.268039 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.267999    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"]
May 26 17:06:15.347240 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.347198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prx4z\" (UniqueName: \"kubernetes.io/projected/a754be44-8031-4393-ae82-46994bc250ae-kube-api-access-prx4z\") pod \"service-ca-operator-57b5b68b8-kkdjh\" (UID: \"a754be44-8031-4393-ae82-46994bc250ae\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.347422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.347314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a754be44-8031-4393-ae82-46994bc250ae-serving-cert\") pod \"service-ca-operator-57b5b68b8-kkdjh\" (UID: \"a754be44-8031-4393-ae82-46994bc250ae\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.347422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.347347    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a754be44-8031-4393-ae82-46994bc250ae-config\") pod \"service-ca-operator-57b5b68b8-kkdjh\" (UID: \"a754be44-8031-4393-ae82-46994bc250ae\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.448681 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.448642    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-prx4z\" (UniqueName: \"kubernetes.io/projected/a754be44-8031-4393-ae82-46994bc250ae-kube-api-access-prx4z\") pod \"service-ca-operator-57b5b68b8-kkdjh\" (UID: \"a754be44-8031-4393-ae82-46994bc250ae\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.448886 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.448726    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a754be44-8031-4393-ae82-46994bc250ae-serving-cert\") pod \"service-ca-operator-57b5b68b8-kkdjh\" (UID: \"a754be44-8031-4393-ae82-46994bc250ae\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.448886 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.448757    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a754be44-8031-4393-ae82-46994bc250ae-config\") pod \"service-ca-operator-57b5b68b8-kkdjh\" (UID: \"a754be44-8031-4393-ae82-46994bc250ae\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.449477 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.449434    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a754be44-8031-4393-ae82-46994bc250ae-config\") pod \"service-ca-operator-57b5b68b8-kkdjh\" (UID: \"a754be44-8031-4393-ae82-46994bc250ae\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.451394 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.451369    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a754be44-8031-4393-ae82-46994bc250ae-serving-cert\") pod \"service-ca-operator-57b5b68b8-kkdjh\" (UID: \"a754be44-8031-4393-ae82-46994bc250ae\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.457699 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.457678    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prx4z\" (UniqueName: \"kubernetes.io/projected/a754be44-8031-4393-ae82-46994bc250ae-kube-api-access-prx4z\") pod \"service-ca-operator-57b5b68b8-kkdjh\" (UID: \"a754be44-8031-4393-ae82-46994bc250ae\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.569190 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.569093    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"
May 26 17:06:15.665386 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.665349    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92"]
May 26 17:06:15.668753 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.668730    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92"
May 26 17:06:15.671554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.671527    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kh7tn\""
May 26 17:06:15.677677 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.677650    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92"]
May 26 17:06:15.751603 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.751559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rcqp\" (UniqueName: \"kubernetes.io/projected/7c05bfdb-3638-40a7-89a5-ab54cb3a6299-kube-api-access-9rcqp\") pod \"network-check-source-7f44b6dd87-gzk92\" (UID: \"7c05bfdb-3638-40a7-89a5-ab54cb3a6299\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92"
May 26 17:06:15.852230 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.852201    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9rcqp\" (UniqueName: \"kubernetes.io/projected/7c05bfdb-3638-40a7-89a5-ab54cb3a6299-kube-api-access-9rcqp\") pod \"network-check-source-7f44b6dd87-gzk92\" (UID: \"7c05bfdb-3638-40a7-89a5-ab54cb3a6299\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92"
May 26 17:06:15.861635 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.861606    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rcqp\" (UniqueName: \"kubernetes.io/projected/7c05bfdb-3638-40a7-89a5-ab54cb3a6299-kube-api-access-9rcqp\") pod \"network-check-source-7f44b6dd87-gzk92\" (UID: \"7c05bfdb-3638-40a7-89a5-ab54cb3a6299\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92"
May 26 17:06:15.893814 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.893781    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh"]
May 26 17:06:15.896726 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:15.896688    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda754be44_8031_4393_ae82_46994bc250ae.slice/crio-8d5b02bf07d9d34184d4acf622e97820505e6af66ce9acfab4bd5649c542b76f WatchSource:0}: Error finding container 8d5b02bf07d9d34184d4acf622e97820505e6af66ce9acfab4bd5649c542b76f: Status 404 returned error can't find the container with id 8d5b02bf07d9d34184d4acf622e97820505e6af66ce9acfab4bd5649c542b76f
May 26 17:06:15.983747 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:15.983706    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92"
May 26 17:06:16.053891 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.053858    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:16.054071 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:16.054046    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 17:06:16.054164 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:16.054130    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls podName:38027300-8fba-4a1d-8f9c-745e9942954a nodeName:}" failed. No retries permitted until 2026-05-26 17:06:24.054108367 +0000 UTC m=+124.779197861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-99qnk" (UID: "38027300-8fba-4a1d-8f9c-745e9942954a") : secret "cluster-monitoring-operator-tls" not found
May 26 17:06:16.107989 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.107912    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92"]
May 26 17:06:16.110794 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:16.110762    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c05bfdb_3638_40a7_89a5_ab54cb3a6299.slice/crio-cdb04a8b5e626110dcbf46e7a461e33b5936d6bf74be9e4d849eaf0ba029280e WatchSource:0}: Error finding container cdb04a8b5e626110dcbf46e7a461e33b5936d6bf74be9e4d849eaf0ba029280e: Status 404 returned error can't find the container with id cdb04a8b5e626110dcbf46e7a461e33b5936d6bf74be9e4d849eaf0ba029280e
May 26 17:06:16.154834 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.154807    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:16.154935 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.154868    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:16.154986 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:16.154973    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 17:06:16.154986 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:16.154980    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:24.154960692 +0000 UTC m=+124.880050185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : configmap references non-existent config key: service-ca.crt
May 26 17:06:16.155098 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:16.155086    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:24.155000116 +0000 UTC m=+124.880089609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : secret "router-metrics-certs-default" not found
May 26 17:06:16.204368 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.204321    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh" event={"ID":"a754be44-8031-4393-ae82-46994bc250ae","Type":"ContainerStarted","Data":"8d5b02bf07d9d34184d4acf622e97820505e6af66ce9acfab4bd5649c542b76f"}
May 26 17:06:16.205775 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.205743    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m" event={"ID":"df3cb020-a879-45c7-8fff-b38a4b5ccf30","Type":"ContainerStarted","Data":"e9e7b570e527504313a069639d29d33fe861fe860a97910855a1e3d528b45e6e"}
May 26 17:06:16.207080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.207058    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92" event={"ID":"7c05bfdb-3638-40a7-89a5-ab54cb3a6299","Type":"ContainerStarted","Data":"3ff1137a8b90d2843a76182860e3d589c59386ece52460b7e95c27ef2b309d50"}
May 26 17:06:16.207214 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.207084    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92" event={"ID":"7c05bfdb-3638-40a7-89a5-ab54cb3a6299","Type":"ContainerStarted","Data":"cdb04a8b5e626110dcbf46e7a461e33b5936d6bf74be9e4d849eaf0ba029280e"}
May 26 17:06:16.222523 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.222479    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m" podStartSLOduration=1.129342815 podStartE2EDuration="3.222463565s" podCreationTimestamp="2026-05-26 17:06:13 +0000 UTC" firstStartedPulling="2026-05-26 17:06:13.708586747 +0000 UTC m=+114.433676226" lastFinishedPulling="2026-05-26 17:06:15.801707483 +0000 UTC m=+116.526796976" observedRunningTime="2026-05-26 17:06:16.222049578 +0000 UTC m=+116.947139081" watchObservedRunningTime="2026-05-26 17:06:16.222463565 +0000 UTC m=+116.947553065"
May 26 17:06:16.236156 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.236110    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-gzk92" podStartSLOduration=1.236096682 podStartE2EDuration="1.236096682s" podCreationTimestamp="2026-05-26 17:06:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:06:16.236028206 +0000 UTC m=+116.961117699" watchObservedRunningTime="2026-05-26 17:06:16.236096682 +0000 UTC m=+116.961186183"
May 26 17:06:16.255554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.255515    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:16.255696 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:16.255676    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 17:06:16.255760 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:16.255750    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls podName:937078be-09eb-4f20-9ef7-eab92c00b8bd nodeName:}" failed. No retries permitted until 2026-05-26 17:06:24.25573073 +0000 UTC m=+124.980820217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls") pod "cluster-samples-operator-697587599-cvqdr" (UID: "937078be-09eb-4f20-9ef7-eab92c00b8bd") : secret "samples-operator-tls" not found
May 26 17:06:16.591279 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.591242    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx"]
May 26 17:06:16.594955 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.594924    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx"
May 26 17:06:16.597658 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.597576    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 26 17:06:16.597658 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.597591    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wx6c9\""
May 26 17:06:16.597658 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.597627    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 26 17:06:16.605933 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.605908    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx"]
May 26 17:06:16.659283 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.659251    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47t9d\" (UniqueName: \"kubernetes.io/projected/dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27-kube-api-access-47t9d\") pod \"migrator-b985f5d89-2d4lx\" (UID: \"dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx"
May 26 17:06:16.760834 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.760798    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-47t9d\" (UniqueName: \"kubernetes.io/projected/dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27-kube-api-access-47t9d\") pod \"migrator-b985f5d89-2d4lx\" (UID: \"dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx"
May 26 17:06:16.769669 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.769642    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47t9d\" (UniqueName: \"kubernetes.io/projected/dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27-kube-api-access-47t9d\") pod \"migrator-b985f5d89-2d4lx\" (UID: \"dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx"
May 26 17:06:16.907753 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:16.907669    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx"
May 26 17:06:17.046828 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:17.046789    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx"]
May 26 17:06:17.051468 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:17.051377    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdfbabf_4c73_46f3_a1f6_bf7cbffbbe27.slice/crio-41e3a1a62a4c818effeafce2a61e3944e502849d50e39bf5c30c1a3b78f1ef3a WatchSource:0}: Error finding container 41e3a1a62a4c818effeafce2a61e3944e502849d50e39bf5c30c1a3b78f1ef3a: Status 404 returned error can't find the container with id 41e3a1a62a4c818effeafce2a61e3944e502849d50e39bf5c30c1a3b78f1ef3a
May 26 17:06:17.210919 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:17.210835    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx" event={"ID":"dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27","Type":"ContainerStarted","Data":"41e3a1a62a4c818effeafce2a61e3944e502849d50e39bf5c30c1a3b78f1ef3a"}
May 26 17:06:19.217305 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:19.217262    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh" event={"ID":"a754be44-8031-4393-ae82-46994bc250ae","Type":"ContainerStarted","Data":"1a905578a2064b3034cee4a641912bafe58ea3c19454724e0a8a211979de055d"}
May 26 17:06:19.218764 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:19.218741    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx" event={"ID":"dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27","Type":"ContainerStarted","Data":"3ba43b6012855747c72de45934c8f5ecd7970b78f4b02db5275b9cca3f110af3"}
May 26 17:06:19.218891 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:19.218768    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx" event={"ID":"dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27","Type":"ContainerStarted","Data":"a3c8db012064e403be5b2b0fab1eb296503f3fd1be9f81f4c4c29620026b0550"}
May 26 17:06:19.235577 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:19.235515    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh" podStartSLOduration=1.49709771 podStartE2EDuration="4.235498664s" podCreationTimestamp="2026-05-26 17:06:15 +0000 UTC" firstStartedPulling="2026-05-26 17:06:15.898554777 +0000 UTC m=+116.623644260" lastFinishedPulling="2026-05-26 17:06:18.636955724 +0000 UTC m=+119.362045214" observedRunningTime="2026-05-26 17:06:19.234985136 +0000 UTC m=+119.960074637" watchObservedRunningTime="2026-05-26 17:06:19.235498664 +0000 UTC m=+119.960588166"
May 26 17:06:19.253328 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:19.253267    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-2d4lx" podStartSLOduration=1.670438246 podStartE2EDuration="3.253247144s" podCreationTimestamp="2026-05-26 17:06:16 +0000 UTC" firstStartedPulling="2026-05-26 17:06:17.053673264 +0000 UTC m=+117.778762746" lastFinishedPulling="2026-05-26 17:06:18.636482154 +0000 UTC m=+119.361571644" observedRunningTime="2026-05-26 17:06:19.251946568 +0000 UTC m=+119.977036069" watchObservedRunningTime="2026-05-26 17:06:19.253247144 +0000 UTC m=+119.978336646"
May 26 17:06:24.121742 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:24.121706    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:24.122111 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:24.121860    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 17:06:24.122111 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:24.121926    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls podName:38027300-8fba-4a1d-8f9c-745e9942954a nodeName:}" failed. No retries permitted until 2026-05-26 17:06:40.121910667 +0000 UTC m=+140.847000152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-99qnk" (UID: "38027300-8fba-4a1d-8f9c-745e9942954a") : secret "cluster-monitoring-operator-tls" not found
May 26 17:06:24.222098 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:24.222062    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:24.222261 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:24.222183    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:24.222261 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:24.222210    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 17:06:24.222339 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:24.222277    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:40.222262091 +0000 UTC m=+140.947351569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : secret "router-metrics-certs-default" not found
May 26 17:06:24.222339 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:24.222291    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle podName:9008dab9-05e5-43b6-bdc8-077476da5900 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:40.222285365 +0000 UTC m=+140.947374844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle") pod "router-default-978fdbd8-97ct5" (UID: "9008dab9-05e5-43b6-bdc8-077476da5900") : configmap references non-existent config key: service-ca.crt
May 26 17:06:24.322525 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:24.322482    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:24.322704 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:24.322635    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 17:06:24.322753 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:24.322708    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls podName:937078be-09eb-4f20-9ef7-eab92c00b8bd nodeName:}" failed. No retries permitted until 2026-05-26 17:06:40.322691566 +0000 UTC m=+141.047781050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls") pod "cluster-samples-operator-697587599-cvqdr" (UID: "937078be-09eb-4f20-9ef7-eab92c00b8bd") : secret "samples-operator-tls" not found
May 26 17:06:29.567818 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:29.567776    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:06:29.568333 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:29.567907    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 17:06:29.568333 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:29.567963    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs podName:a13dbe45-e72d-4315-a2a1-c4736202fd9b nodeName:}" failed. No retries permitted until 2026-05-26 17:08:31.567948281 +0000 UTC m=+252.293037764 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs") pod "network-metrics-daemon-lztjw" (UID: "a13dbe45-e72d-4315-a2a1-c4736202fd9b") : secret "metrics-daemon-secret" not found
May 26 17:06:40.156388 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.156350    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:40.158798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.158778    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38027300-8fba-4a1d-8f9c-745e9942954a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-99qnk\" (UID: \"38027300-8fba-4a1d-8f9c-745e9942954a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:40.257622 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.257576    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:40.257879 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.257642    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:40.258226 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.258203    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9008dab9-05e5-43b6-bdc8-077476da5900-service-ca-bundle\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:40.259971 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.259953    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9008dab9-05e5-43b6-bdc8-077476da5900-metrics-certs\") pod \"router-default-978fdbd8-97ct5\" (UID: \"9008dab9-05e5-43b6-bdc8-077476da5900\") " pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:40.358944 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.358901    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:40.361286 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.361262    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/937078be-09eb-4f20-9ef7-eab92c00b8bd-samples-operator-tls\") pod \"cluster-samples-operator-697587599-cvqdr\" (UID: \"937078be-09eb-4f20-9ef7-eab92c00b8bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:40.382686 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.382665    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tjfkb\""
May 26 17:06:40.390817 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.390797    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"
May 26 17:06:40.499710 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.499682    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-96mvf\""
May 26 17:06:40.507847 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.507828    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:40.511998 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.511921    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk"]
May 26 17:06:40.514849 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:40.514821    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38027300_8fba_4a1d_8f9c_745e9942954a.slice/crio-32676ca91d85acec5e3ddf3f66df310909ed253d35846ebfc64956ba1a5d57f0 WatchSource:0}: Error finding container 32676ca91d85acec5e3ddf3f66df310909ed253d35846ebfc64956ba1a5d57f0: Status 404 returned error can't find the container with id 32676ca91d85acec5e3ddf3f66df310909ed253d35846ebfc64956ba1a5d57f0
May 26 17:06:40.586820 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.586793    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6wdf9\""
May 26 17:06:40.594210 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.594130    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"
May 26 17:06:40.643329 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.643299    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-978fdbd8-97ct5"]
May 26 17:06:40.648248 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:40.648212    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9008dab9_05e5_43b6_bdc8_077476da5900.slice/crio-38964db603024a3f0a2be6432e9e5c1354247f691cf1ec847f30f3d67a14bead WatchSource:0}: Error finding container 38964db603024a3f0a2be6432e9e5c1354247f691cf1ec847f30f3d67a14bead: Status 404 returned error can't find the container with id 38964db603024a3f0a2be6432e9e5c1354247f691cf1ec847f30f3d67a14bead
May 26 17:06:40.717841 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:40.717811    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr"]
May 26 17:06:41.272493 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:41.272433    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-978fdbd8-97ct5" event={"ID":"9008dab9-05e5-43b6-bdc8-077476da5900","Type":"ContainerStarted","Data":"0a586bcf2f84d035dc3d3ced1a3c8bd04b9bf267312efa447aa2de359e4ee19d"}
May 26 17:06:41.272493 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:41.272476    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-978fdbd8-97ct5" event={"ID":"9008dab9-05e5-43b6-bdc8-077476da5900","Type":"ContainerStarted","Data":"38964db603024a3f0a2be6432e9e5c1354247f691cf1ec847f30f3d67a14bead"}
May 26 17:06:41.273785 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:41.273752    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk" event={"ID":"38027300-8fba-4a1d-8f9c-745e9942954a","Type":"ContainerStarted","Data":"32676ca91d85acec5e3ddf3f66df310909ed253d35846ebfc64956ba1a5d57f0"}
May 26 17:06:41.274722 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:41.274694    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr" event={"ID":"937078be-09eb-4f20-9ef7-eab92c00b8bd","Type":"ContainerStarted","Data":"6844ae8f898dc933cb1d0b0a6fe5e364832406787f97cc6d9098773cf7a1556e"}
May 26 17:06:41.290729 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:41.290682    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-978fdbd8-97ct5" podStartSLOduration=33.290666383 podStartE2EDuration="33.290666383s" podCreationTimestamp="2026-05-26 17:06:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:06:41.289565002 +0000 UTC m=+142.014654504" watchObservedRunningTime="2026-05-26 17:06:41.290666383 +0000 UTC m=+142.015755884"
May 26 17:06:41.509095 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:41.509057    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:41.512198 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:41.512167    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:42.278071 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:42.278039    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:42.279555 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:42.279528    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-978fdbd8-97ct5"
May 26 17:06:43.072026 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.071983    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tk2sx"]
May 26 17:06:43.074208 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.074189    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.077313 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.077286    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ngxff\""
May 26 17:06:43.078219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.078204    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 17:06:43.078292 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.078265    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 17:06:43.080617 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.080549    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fb0788f-2272-49e4-8f3f-ca83c46f2004-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.080700 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.080632    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md8ct\" (UniqueName: \"kubernetes.io/projected/2fb0788f-2272-49e4-8f3f-ca83c46f2004-kube-api-access-md8ct\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.080700 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.080669    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fb0788f-2272-49e4-8f3f-ca83c46f2004-data-volume\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.080794 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.080704    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fb0788f-2272-49e4-8f3f-ca83c46f2004-crio-socket\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.080794 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.080734    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fb0788f-2272-49e4-8f3f-ca83c46f2004-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.085283 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.085255    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tk2sx"]
May 26 17:06:43.181536 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.181449    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fb0788f-2272-49e4-8f3f-ca83c46f2004-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.181536 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.181525    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-md8ct\" (UniqueName: \"kubernetes.io/projected/2fb0788f-2272-49e4-8f3f-ca83c46f2004-kube-api-access-md8ct\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.181738 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.181558    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fb0788f-2272-49e4-8f3f-ca83c46f2004-data-volume\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.181738 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.181589    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fb0788f-2272-49e4-8f3f-ca83c46f2004-crio-socket\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.181738 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.181615    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fb0788f-2272-49e4-8f3f-ca83c46f2004-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.181738 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.181700    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fb0788f-2272-49e4-8f3f-ca83c46f2004-crio-socket\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.181977 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.181957    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fb0788f-2272-49e4-8f3f-ca83c46f2004-data-volume\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.182215 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.182196    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fb0788f-2272-49e4-8f3f-ca83c46f2004-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.184117 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.184098    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fb0788f-2272-49e4-8f3f-ca83c46f2004-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.199218 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.199105    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md8ct\" (UniqueName: \"kubernetes.io/projected/2fb0788f-2272-49e4-8f3f-ca83c46f2004-kube-api-access-md8ct\") pod \"insights-runtime-extractor-tk2sx\" (UID: \"2fb0788f-2272-49e4-8f3f-ca83c46f2004\") " pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.283706 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.283672    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk" event={"ID":"38027300-8fba-4a1d-8f9c-745e9942954a","Type":"ContainerStarted","Data":"c1d346521a555f73f6332ce86c66f472a7c1e6dba582912f757462f67ea1c3fb"}
May 26 17:06:43.285421 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.285386    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr" event={"ID":"937078be-09eb-4f20-9ef7-eab92c00b8bd","Type":"ContainerStarted","Data":"5aaad75a1b8854087e44c89dfbdf8534b62b18be6c2270a6616215ce8e772f59"}
May 26 17:06:43.285421 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.285422    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr" event={"ID":"937078be-09eb-4f20-9ef7-eab92c00b8bd","Type":"ContainerStarted","Data":"470d00e5442a2d19220d3bcec0c29fd0a7bf559222d301862d9eacf094f80bc9"}
May 26 17:06:43.317231 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.317172    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-99qnk" podStartSLOduration=32.979757294 podStartE2EDuration="35.317155771s" podCreationTimestamp="2026-05-26 17:06:08 +0000 UTC" firstStartedPulling="2026-05-26 17:06:40.516634989 +0000 UTC m=+141.241724468" lastFinishedPulling="2026-05-26 17:06:42.854033461 +0000 UTC m=+143.579122945" observedRunningTime="2026-05-26 17:06:43.316204719 +0000 UTC m=+144.041294221" watchObservedRunningTime="2026-05-26 17:06:43.317155771 +0000 UTC m=+144.042245273"
May 26 17:06:43.336998 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.336945    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-cvqdr" podStartSLOduration=33.245252751 podStartE2EDuration="35.336931464s" podCreationTimestamp="2026-05-26 17:06:08 +0000 UTC" firstStartedPulling="2026-05-26 17:06:40.763978623 +0000 UTC m=+141.489068103" lastFinishedPulling="2026-05-26 17:06:42.855657319 +0000 UTC m=+143.580746816" observedRunningTime="2026-05-26 17:06:43.336427114 +0000 UTC m=+144.061516792" watchObservedRunningTime="2026-05-26 17:06:43.336931464 +0000 UTC m=+144.062020965"
May 26 17:06:43.383200 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.383169    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tk2sx"
May 26 17:06:43.463271 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.462821    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"]
May 26 17:06:43.471959 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.471935    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"
May 26 17:06:43.474927 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.474743    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2p9f5\""
May 26 17:06:43.474927 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.474769    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 26 17:06:43.474927 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.474813    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"]
May 26 17:06:43.484100 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.484076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9d4b70a-fb28-4c67-a645-07307550e2db-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs\" (UID: \"d9d4b70a-fb28-4c67-a645-07307550e2db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"
May 26 17:06:43.512945 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.512918    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tk2sx"]
May 26 17:06:43.515870 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:43.515845    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb0788f_2272_49e4_8f3f_ca83c46f2004.slice/crio-e71590b11373be86052d2ea01d3540d65495f030e147fc884413afd9095d2fe0 WatchSource:0}: Error finding container e71590b11373be86052d2ea01d3540d65495f030e147fc884413afd9095d2fe0: Status 404 returned error can't find the container with id e71590b11373be86052d2ea01d3540d65495f030e147fc884413afd9095d2fe0
May 26 17:06:43.585225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:43.585198    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9d4b70a-fb28-4c67-a645-07307550e2db-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs\" (UID: \"d9d4b70a-fb28-4c67-a645-07307550e2db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"
May 26 17:06:43.585395 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:43.585329    2571 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 26 17:06:43.585395 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:43.585394    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d4b70a-fb28-4c67-a645-07307550e2db-tls-certificates podName:d9d4b70a-fb28-4c67-a645-07307550e2db nodeName:}" failed. No retries permitted until 2026-05-26 17:06:44.085376534 +0000 UTC m=+144.810466016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/d9d4b70a-fb28-4c67-a645-07307550e2db-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs" (UID: "d9d4b70a-fb28-4c67-a645-07307550e2db") : secret "prometheus-operator-admission-webhook-tls" not found
May 26 17:06:44.089924 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:44.089871    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9d4b70a-fb28-4c67-a645-07307550e2db-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs\" (UID: \"d9d4b70a-fb28-4c67-a645-07307550e2db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"
May 26 17:06:44.092642 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:44.092607    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9d4b70a-fb28-4c67-a645-07307550e2db-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs\" (UID: \"d9d4b70a-fb28-4c67-a645-07307550e2db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"
May 26 17:06:44.289652 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:44.289623    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tk2sx" event={"ID":"2fb0788f-2272-49e4-8f3f-ca83c46f2004","Type":"ContainerStarted","Data":"0533e1fe5ac1afb93b89a6e482c6536266baed8c301221ca2fa5b9f40f8ae46b"}
May 26 17:06:44.290054 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:44.289661    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tk2sx" event={"ID":"2fb0788f-2272-49e4-8f3f-ca83c46f2004","Type":"ContainerStarted","Data":"93d5e0403aa834a7c98907044d3cf80795bd5037ee1eba2fc81a092a8ad30411"}
May 26 17:06:44.290054 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:44.289673    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tk2sx" event={"ID":"2fb0788f-2272-49e4-8f3f-ca83c46f2004","Type":"ContainerStarted","Data":"e71590b11373be86052d2ea01d3540d65495f030e147fc884413afd9095d2fe0"}
May 26 17:06:44.384561 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:44.384481    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"
May 26 17:06:44.517993 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:44.517969    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"]
May 26 17:06:44.520266 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:44.520226    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d4b70a_fb28_4c67_a645_07307550e2db.slice/crio-2da82921010b49e035b5814c6e9aeaaba481fdcbae5ce934917cb90b0ca48af4 WatchSource:0}: Error finding container 2da82921010b49e035b5814c6e9aeaaba481fdcbae5ce934917cb90b0ca48af4: Status 404 returned error can't find the container with id 2da82921010b49e035b5814c6e9aeaaba481fdcbae5ce934917cb90b0ca48af4
May 26 17:06:45.293743 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:45.293698    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs" event={"ID":"d9d4b70a-fb28-4c67-a645-07307550e2db","Type":"ContainerStarted","Data":"2da82921010b49e035b5814c6e9aeaaba481fdcbae5ce934917cb90b0ca48af4"}
May 26 17:06:46.298083 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:46.298046    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs" event={"ID":"d9d4b70a-fb28-4c67-a645-07307550e2db","Type":"ContainerStarted","Data":"d74d2d4e1ef807ac165e434c803ca16075f6d0e602c45c2117f4711ad6388ed6"}
May 26 17:06:46.298575 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:46.298239    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"
May 26 17:06:46.300172 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:46.300137    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tk2sx" event={"ID":"2fb0788f-2272-49e4-8f3f-ca83c46f2004","Type":"ContainerStarted","Data":"112a99efb9b32e9f2ae5264ff1ea071bce6699eb2c6b54045f8c8facc05b32a4"}
May 26 17:06:46.303162 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:46.303142    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs"
May 26 17:06:46.316180 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:46.316140    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs" podStartSLOduration=1.762890973 podStartE2EDuration="3.316129077s" podCreationTimestamp="2026-05-26 17:06:43 +0000 UTC" firstStartedPulling="2026-05-26 17:06:44.522226739 +0000 UTC m=+145.247316218" lastFinishedPulling="2026-05-26 17:06:46.075464829 +0000 UTC m=+146.800554322" observedRunningTime="2026-05-26 17:06:46.314523619 +0000 UTC m=+147.039613121" watchObservedRunningTime="2026-05-26 17:06:46.316129077 +0000 UTC m=+147.041218598"
May 26 17:06:46.331328 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:46.331244    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tk2sx" podStartSLOduration=0.820915226 podStartE2EDuration="3.331233109s" podCreationTimestamp="2026-05-26 17:06:43 +0000 UTC" firstStartedPulling="2026-05-26 17:06:43.568321044 +0000 UTC m=+144.293410523" lastFinishedPulling="2026-05-26 17:06:46.078638922 +0000 UTC m=+146.803728406" observedRunningTime="2026-05-26 17:06:46.329644402 +0000 UTC m=+147.054733902" watchObservedRunningTime="2026-05-26 17:06:46.331233109 +0000 UTC m=+147.056322610"
May 26 17:06:50.868840 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.868802    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"]
May 26 17:06:50.871708 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.871683    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:50.874658 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.874635    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 26 17:06:50.874778 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.874682    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 26 17:06:50.875902 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.875883    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d8xxz\""
May 26 17:06:50.875991 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.875931    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 17:06:50.886468 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.886447    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"]
May 26 17:06:50.888767 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.888746    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"]
May 26 17:06:50.891322 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.891304    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:50.892920 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.892897    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nrhrc"]
May 26 17:06:50.894194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.894172    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 26 17:06:50.894303 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.894172    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 26 17:06:50.894575 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.894553    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 26 17:06:50.895582 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.895566    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x2jn2\""
May 26 17:06:50.895659 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.895588    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:50.898424 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.898399    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 17:06:50.898638 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.898541    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 17:06:50.898701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.898640    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2j828\""
May 26 17:06:50.898918 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.898886    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 17:06:50.904799 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.904773    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"]
May 26 17:06:50.949057 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949004    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-wtmp\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:50.949239 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949070    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:50.949239 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949115    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08843935-911b-4d43-ad28-f7c15f527321-metrics-client-ca\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:50.949239 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:50.949239 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949190    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggp5\" (UniqueName: \"kubernetes.io/projected/08843935-911b-4d43-ad28-f7c15f527321-kube-api-access-5ggp5\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:50.949418 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-tls\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:50.949418 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949331    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:50.949418 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949367    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2wt2\" (UniqueName: \"kubernetes.io/projected/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-api-access-m2wt2\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:50.949546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949459    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/089437a0-9dc8-4425-9660-bbbc7e10ab68-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:50.949546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/089437a0-9dc8-4425-9660-bbbc7e10ab68-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:50.949546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949533    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:50.949690 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949549    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:50.949690 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949565    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkvh\" (UniqueName: \"kubernetes.io/projected/089437a0-9dc8-4425-9660-bbbc7e10ab68-kube-api-access-zpkvh\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:50.949690 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949628    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/089437a0-9dc8-4425-9660-bbbc7e10ab68-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:50.949690 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949660    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08843935-911b-4d43-ad28-f7c15f527321-root\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:50.949690 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949685    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-textfile\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:50.949917 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949716    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08843935-911b-4d43-ad28-f7c15f527321-sys\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:50.949917 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:50.949917 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:50.949778    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050668 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050635    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08843935-911b-4d43-ad28-f7c15f527321-root\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050668 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050674    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-textfile\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050696    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08843935-911b-4d43-ad28-f7c15f527321-sys\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050714    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050739    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050752    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08843935-911b-4d43-ad28-f7c15f527321-root\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050774    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-wtmp\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050779    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08843935-911b-4d43-ad28-f7c15f527321-sys\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050828    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.050901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050871    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08843935-911b-4d43-ad28-f7c15f527321-metrics-client-ca\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.050901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050898    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050931    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggp5\" (UniqueName: \"kubernetes.io/projected/08843935-911b-4d43-ad28-f7c15f527321-kube-api-access-5ggp5\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050961    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-tls\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050964    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-wtmp\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.050993    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051046    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m2wt2\" (UniqueName: \"kubernetes.io/projected/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-api-access-m2wt2\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051101    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/089437a0-9dc8-4425-9660-bbbc7e10ab68-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051137    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/089437a0-9dc8-4425-9660-bbbc7e10ab68-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051144    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-textfile\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051177    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051203    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051229    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkvh\" (UniqueName: \"kubernetes.io/projected/089437a0-9dc8-4425-9660-bbbc7e10ab68-kube-api-access-zpkvh\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051261    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/089437a0-9dc8-4425-9660-bbbc7e10ab68-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:51.051331 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051268    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.051956 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:51.051382    2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 17:06:51.051956 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:51.051440    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-tls podName:08843935-911b-4d43-ad28-f7c15f527321 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:51.551421204 +0000 UTC m=+152.276510687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-tls") pod "node-exporter-nrhrc" (UID: "08843935-911b-4d43-ad28-f7c15f527321") : secret "node-exporter-tls" not found
May 26 17:06:51.051956 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.051707    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08843935-911b-4d43-ad28-f7c15f527321-metrics-client-ca\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.052152 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.052072    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/089437a0-9dc8-4425-9660-bbbc7e10ab68-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:51.052325 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.052299    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.052733 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.052712    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.052872 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.052732    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.053720 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.053694    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.053841 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.053819    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.054127 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.054108    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/089437a0-9dc8-4425-9660-bbbc7e10ab68-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:51.054198 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.054107    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.054269 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.054236    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/089437a0-9dc8-4425-9660-bbbc7e10ab68-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:51.059647 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.059622    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2wt2\" (UniqueName: \"kubernetes.io/projected/ae21fdd2-8056-4641-9dd2-d0ef955e4a0a-kube-api-access-m2wt2\") pod \"kube-state-metrics-54fd6bc5c4-mfcrs\" (UID: \"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.059647 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.059639    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggp5\" (UniqueName: \"kubernetes.io/projected/08843935-911b-4d43-ad28-f7c15f527321-kube-api-access-5ggp5\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.060553 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.060533    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkvh\" (UniqueName: \"kubernetes.io/projected/089437a0-9dc8-4425-9660-bbbc7e10ab68-kube-api-access-zpkvh\") pod \"openshift-state-metrics-9c959559f-8fhl2\" (UID: \"089437a0-9dc8-4425-9660-bbbc7e10ab68\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:51.180812 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.180733    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"
May 26 17:06:51.202054 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.202005    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"
May 26 17:06:51.316853 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.316818    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2"]
May 26 17:06:51.319703 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:51.319670    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089437a0_9dc8_4425_9660_bbbc7e10ab68.slice/crio-57a6d80efdd75d01333d0d14de0c10f54a44fc318572e2783f5a66c33508b7bb WatchSource:0}: Error finding container 57a6d80efdd75d01333d0d14de0c10f54a44fc318572e2783f5a66c33508b7bb: Status 404 returned error can't find the container with id 57a6d80efdd75d01333d0d14de0c10f54a44fc318572e2783f5a66c33508b7bb
May 26 17:06:51.336232 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.336204    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs"]
May 26 17:06:51.339006 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:51.338982    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae21fdd2_8056_4641_9dd2_d0ef955e4a0a.slice/crio-040624f645d4be5f2f123786b7edb889826f3f74832844972fa237805870f814 WatchSource:0}: Error finding container 040624f645d4be5f2f123786b7edb889826f3f74832844972fa237805870f814: Status 404 returned error can't find the container with id 040624f645d4be5f2f123786b7edb889826f3f74832844972fa237805870f814
May 26 17:06:51.555503 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.555465    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-tls\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.557779 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.557757    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08843935-911b-4d43-ad28-f7c15f527321-node-exporter-tls\") pod \"node-exporter-nrhrc\" (UID: \"08843935-911b-4d43-ad28-f7c15f527321\") " pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.807875 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:51.807845    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrhrc"
May 26 17:06:51.818500 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:51.818459    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08843935_911b_4d43_ad28_f7c15f527321.slice/crio-0e519380ae2cdd7da22c2d001ecdd732d3423c04fb667234b14ef13b44f50257 WatchSource:0}: Error finding container 0e519380ae2cdd7da22c2d001ecdd732d3423c04fb667234b14ef13b44f50257: Status 404 returned error can't find the container with id 0e519380ae2cdd7da22c2d001ecdd732d3423c04fb667234b14ef13b44f50257
May 26 17:06:52.322622 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.322580    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrhrc" event={"ID":"08843935-911b-4d43-ad28-f7c15f527321","Type":"ContainerStarted","Data":"0e519380ae2cdd7da22c2d001ecdd732d3423c04fb667234b14ef13b44f50257"}
May 26 17:06:52.323801 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.323771    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs" event={"ID":"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a","Type":"ContainerStarted","Data":"040624f645d4be5f2f123786b7edb889826f3f74832844972fa237805870f814"}
May 26 17:06:52.325615 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.325588    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2" event={"ID":"089437a0-9dc8-4425-9660-bbbc7e10ab68","Type":"ContainerStarted","Data":"ed107c9aef78a77609933329ba63e42f9596bc9dbac0c398f3ee4135f6757c48"}
May 26 17:06:52.325615 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.325618    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2" event={"ID":"089437a0-9dc8-4425-9660-bbbc7e10ab68","Type":"ContainerStarted","Data":"0ceab79566d3577137263c192d298447a6825fe83163a83f554dcd01d423b284"}
May 26 17:06:52.325786 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.325627    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2" event={"ID":"089437a0-9dc8-4425-9660-bbbc7e10ab68","Type":"ContainerStarted","Data":"57a6d80efdd75d01333d0d14de0c10f54a44fc318572e2783f5a66c33508b7bb"}
May 26 17:06:52.842378 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.842317    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"]
May 26 17:06:52.846141 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.846112    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:52.849071 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.849044    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 26 17:06:52.849213 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.849096    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qqx9h\""
May 26 17:06:52.849213 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.849104    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 26 17:06:52.849332 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.849304    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 26 17:06:52.849384 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.849356    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 26 17:06:52.849472 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.849449    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 26 17:06:52.849555 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.849535    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c85p446o8a41p\""
May 26 17:06:52.856049 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.856005    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"]
May 26 17:06:52.970218 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.970178    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:52.970218 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.970224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:52.970461 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.970313    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2wtq\" (UniqueName: \"kubernetes.io/projected/e77b3862-aab3-485a-8910-bfa16b5c9050-kube-api-access-x2wtq\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:52.970461 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.970350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-grpc-tls\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:52.970461 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.970380    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e77b3862-aab3-485a-8910-bfa16b5c9050-metrics-client-ca\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:52.970461 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.970412    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:52.970615 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.970521    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:52.970615 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:52.970550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-tls\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.071078 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.071051    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x2wtq\" (UniqueName: \"kubernetes.io/projected/e77b3862-aab3-485a-8910-bfa16b5c9050-kube-api-access-x2wtq\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.071215 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.071094    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-grpc-tls\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.071281 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.071231    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e77b3862-aab3-485a-8910-bfa16b5c9050-metrics-client-ca\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.071336 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.071278    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.071389 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.071341    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.071389 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.071359    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-tls\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.071492 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.071390    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.071492 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.071422    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.072107 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.072047    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e77b3862-aab3-485a-8910-bfa16b5c9050-metrics-client-ca\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.074345 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.074307    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.074345 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.074337    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-tls\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.074527 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.074427    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-grpc-tls\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.074589 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.074564    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.075099 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.074889    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.075286 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.075263    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e77b3862-aab3-485a-8910-bfa16b5c9050-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.079866 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.079844    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2wtq\" (UniqueName: \"kubernetes.io/projected/e77b3862-aab3-485a-8910-bfa16b5c9050-kube-api-access-x2wtq\") pod \"thanos-querier-7dc4d8cdc8-p7m7f\" (UID: \"e77b3862-aab3-485a-8910-bfa16b5c9050\") " pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.158810 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.158371    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:53.330118 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.330088    2571 generic.go:358] "Generic (PLEG): container finished" podID="08843935-911b-4d43-ad28-f7c15f527321" containerID="ccfee2b41d07376c009f50b55c180d93a9ed27145a2c6ab5a8a48643c8242455" exitCode=0
May 26 17:06:53.330447 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.330188    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrhrc" event={"ID":"08843935-911b-4d43-ad28-f7c15f527321","Type":"ContainerDied","Data":"ccfee2b41d07376c009f50b55c180d93a9ed27145a2c6ab5a8a48643c8242455"}
May 26 17:06:53.333517 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.333489    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs" event={"ID":"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a","Type":"ContainerStarted","Data":"0a999bd8749869e6f53f19448565c39d0f2d40b8e180541fc9d983b2f68601dc"}
May 26 17:06:53.333639 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.333523    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs" event={"ID":"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a","Type":"ContainerStarted","Data":"f8b3e1df468e288edf379a3018255b033d2ffbe5234a4ecb4add22be7e82a004"}
May 26 17:06:53.333639 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.333533    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs" event={"ID":"ae21fdd2-8056-4641-9dd2-d0ef955e4a0a","Type":"ContainerStarted","Data":"39869b72100f57423b34ddabd4021616578ff0c432c73ddd03cb39cff9c27bde"}
May 26 17:06:53.335780 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.335757    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2" event={"ID":"089437a0-9dc8-4425-9660-bbbc7e10ab68","Type":"ContainerStarted","Data":"47a83ada180893812287e90cb3e6673ae5a4c9abbbee7c35d249b2ef981f374d"}
May 26 17:06:53.341058 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.341007    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"]
May 26 17:06:53.344703 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:53.344678    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77b3862_aab3_485a_8910_bfa16b5c9050.slice/crio-2683525d9eb87e1e3df01f5395231d82295d508bdabd6695bd06958c40f9e77d WatchSource:0}: Error finding container 2683525d9eb87e1e3df01f5395231d82295d508bdabd6695bd06958c40f9e77d: Status 404 returned error can't find the container with id 2683525d9eb87e1e3df01f5395231d82295d508bdabd6695bd06958c40f9e77d
May 26 17:06:53.367662 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.367569    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mfcrs" podStartSLOduration=1.685880549 podStartE2EDuration="3.367550887s" podCreationTimestamp="2026-05-26 17:06:50 +0000 UTC" firstStartedPulling="2026-05-26 17:06:51.340646455 +0000 UTC m=+152.065735934" lastFinishedPulling="2026-05-26 17:06:53.022316785 +0000 UTC m=+153.747406272" observedRunningTime="2026-05-26 17:06:53.366606036 +0000 UTC m=+154.091695536" watchObservedRunningTime="2026-05-26 17:06:53.367550887 +0000 UTC m=+154.092640392"
May 26 17:06:53.384676 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:53.384634    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8fhl2" podStartSLOduration=1.7993131180000002 podStartE2EDuration="3.384619918s" podCreationTimestamp="2026-05-26 17:06:50 +0000 UTC" firstStartedPulling="2026-05-26 17:06:51.437136922 +0000 UTC m=+152.162226401" lastFinishedPulling="2026-05-26 17:06:53.022443711 +0000 UTC m=+153.747533201" observedRunningTime="2026-05-26 17:06:53.382787911 +0000 UTC m=+154.107877413" watchObservedRunningTime="2026-05-26 17:06:53.384619918 +0000 UTC m=+154.109709483"
May 26 17:06:54.340039 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:54.339906    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f" event={"ID":"e77b3862-aab3-485a-8910-bfa16b5c9050","Type":"ContainerStarted","Data":"2683525d9eb87e1e3df01f5395231d82295d508bdabd6695bd06958c40f9e77d"}
May 26 17:06:54.341758 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:54.341728    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrhrc" event={"ID":"08843935-911b-4d43-ad28-f7c15f527321","Type":"ContainerStarted","Data":"53efccccb02f8acd042b907c5a2866456e3cb986dca6185817e5a150587b9a90"}
May 26 17:06:54.341894 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:54.341766    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrhrc" event={"ID":"08843935-911b-4d43-ad28-f7c15f527321","Type":"ContainerStarted","Data":"fb2acb45de0d99d1861fc5b86ebd13427ddd9db648b90e319b023a33b9ffe422"}
May 26 17:06:54.363864 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:54.363804    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nrhrc" podStartSLOduration=3.159284762 podStartE2EDuration="4.363789929s" podCreationTimestamp="2026-05-26 17:06:50 +0000 UTC" firstStartedPulling="2026-05-26 17:06:51.820635124 +0000 UTC m=+152.545724603" lastFinishedPulling="2026-05-26 17:06:53.025140281 +0000 UTC m=+153.750229770" observedRunningTime="2026-05-26 17:06:54.361305554 +0000 UTC m=+155.086395081" watchObservedRunningTime="2026-05-26 17:06:54.363789929 +0000 UTC m=+155.088879429"
May 26 17:06:55.279676 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.277576    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"]
May 26 17:06:55.281137 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.281110    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.284031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.283984    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-t9jf2\""
May 26 17:06:55.285167 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.285147    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 26 17:06:55.285269 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.285230    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9d9ev3m5vnb1i\""
May 26 17:06:55.285477 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.285457    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 26 17:06:55.285647 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.285461    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 26 17:06:55.285770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.285496    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 26 17:06:55.288615 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.288593    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"]
May 26 17:06:55.393291 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.393217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8ef69f78-f149-4702-b560-d1a103725d7b-secret-metrics-server-client-certs\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.393866 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.393306    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8ef69f78-f149-4702-b560-d1a103725d7b-client-ca-bundle\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.393866 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.393365    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26bnw\" (UniqueName: \"kubernetes.io/projected/8ef69f78-f149-4702-b560-d1a103725d7b-kube-api-access-26bnw\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.393866 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.393404    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ef69f78-f149-4702-b560-d1a103725d7b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.393866 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.393474    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8ef69f78-f149-4702-b560-d1a103725d7b-audit-log\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.393866 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.393507    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8ef69f78-f149-4702-b560-d1a103725d7b-secret-metrics-server-tls\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.393866 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.393612    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8ef69f78-f149-4702-b560-d1a103725d7b-metrics-server-audit-profiles\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.495030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.494987    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8ef69f78-f149-4702-b560-d1a103725d7b-audit-log\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.495030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.495037    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8ef69f78-f149-4702-b560-d1a103725d7b-secret-metrics-server-tls\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.495242 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.495167    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8ef69f78-f149-4702-b560-d1a103725d7b-metrics-server-audit-profiles\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.495305 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.495279    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8ef69f78-f149-4702-b560-d1a103725d7b-secret-metrics-server-client-certs\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.495363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.495322    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8ef69f78-f149-4702-b560-d1a103725d7b-client-ca-bundle\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.495419 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.495357    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-26bnw\" (UniqueName: \"kubernetes.io/projected/8ef69f78-f149-4702-b560-d1a103725d7b-kube-api-access-26bnw\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.495419 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.495379    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8ef69f78-f149-4702-b560-d1a103725d7b-audit-log\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.495419 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.495399    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ef69f78-f149-4702-b560-d1a103725d7b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.496118 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.496092    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8ef69f78-f149-4702-b560-d1a103725d7b-metrics-server-audit-profiles\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.496252 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.496130    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ef69f78-f149-4702-b560-d1a103725d7b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.497705 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.497673    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8ef69f78-f149-4702-b560-d1a103725d7b-secret-metrics-server-tls\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.497824 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.497778    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8ef69f78-f149-4702-b560-d1a103725d7b-client-ca-bundle\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.498046 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.498023    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8ef69f78-f149-4702-b560-d1a103725d7b-secret-metrics-server-client-certs\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.503669 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.503646    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26bnw\" (UniqueName: \"kubernetes.io/projected/8ef69f78-f149-4702-b560-d1a103725d7b-kube-api-access-26bnw\") pod \"metrics-server-5d8d68d57b-m8qpx\" (UID: \"8ef69f78-f149-4702-b560-d1a103725d7b\") " pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.593089 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.593057    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:06:55.636201 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.636170    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"]
May 26 17:06:55.639284 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.639263    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"
May 26 17:06:55.641845 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.641821    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-rcbbw\""
May 26 17:06:55.641995 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.641983    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 26 17:06:55.648225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.648171    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"]
May 26 17:06:55.699193 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.699149    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/380972e0-8864-4ad5-8e16-85ce0e8a34b2-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-d5zwx\" (UID: \"380972e0-8864-4ad5-8e16-85ce0e8a34b2\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"
May 26 17:06:55.740736 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.740706    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"]
May 26 17:06:55.743635 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:55.743608    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef69f78_f149_4702_b560_d1a103725d7b.slice/crio-a8149e88e9b2e14f4bfb887ae6b88a0837ba8fbb547b1cbc229a0f052b952f70 WatchSource:0}: Error finding container a8149e88e9b2e14f4bfb887ae6b88a0837ba8fbb547b1cbc229a0f052b952f70: Status 404 returned error can't find the container with id a8149e88e9b2e14f4bfb887ae6b88a0837ba8fbb547b1cbc229a0f052b952f70
May 26 17:06:55.799995 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:55.799902    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/380972e0-8864-4ad5-8e16-85ce0e8a34b2-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-d5zwx\" (UID: \"380972e0-8864-4ad5-8e16-85ce0e8a34b2\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"
May 26 17:06:55.800195 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:55.800097    2571 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 26 17:06:55.800195 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:55.800176    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380972e0-8864-4ad5-8e16-85ce0e8a34b2-monitoring-plugin-cert podName:380972e0-8864-4ad5-8e16-85ce0e8a34b2 nodeName:}" failed. No retries permitted until 2026-05-26 17:06:56.300155472 +0000 UTC m=+157.025244965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/380972e0-8864-4ad5-8e16-85ce0e8a34b2-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-d5zwx" (UID: "380972e0-8864-4ad5-8e16-85ce0e8a34b2") : secret "monitoring-plugin-cert" not found
May 26 17:06:56.305485 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:56.305450    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/380972e0-8864-4ad5-8e16-85ce0e8a34b2-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-d5zwx\" (UID: \"380972e0-8864-4ad5-8e16-85ce0e8a34b2\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"
May 26 17:06:56.308094 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:56.308069    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/380972e0-8864-4ad5-8e16-85ce0e8a34b2-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-d5zwx\" (UID: \"380972e0-8864-4ad5-8e16-85ce0e8a34b2\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"
May 26 17:06:56.350743 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:56.350708    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f" event={"ID":"e77b3862-aab3-485a-8910-bfa16b5c9050","Type":"ContainerStarted","Data":"b0c971b8d50ca19d469079444f2f78dd49a88f60839968a8fce9f0a602e3a372"}
May 26 17:06:56.350743 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:56.350748    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f" event={"ID":"e77b3862-aab3-485a-8910-bfa16b5c9050","Type":"ContainerStarted","Data":"faabfc60fdad61ebd40b103f4d1d732ea275483568bbcdb6a7e1363797c56cf0"}
May 26 17:06:56.350972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:56.350766    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f" event={"ID":"e77b3862-aab3-485a-8910-bfa16b5c9050","Type":"ContainerStarted","Data":"d55cbf6fef730729ae81d3222c76c3f11836ea05c9fc808828d9520a716c4db8"}
May 26 17:06:56.351895 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:56.351859    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx" event={"ID":"8ef69f78-f149-4702-b560-d1a103725d7b","Type":"ContainerStarted","Data":"a8149e88e9b2e14f4bfb887ae6b88a0837ba8fbb547b1cbc229a0f052b952f70"}
May 26 17:06:56.559205 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:56.559127    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"
May 26 17:06:56.725960 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:56.725823    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4c8fb" podUID="91c3c9fd-a70c-4e23-85db-d5544f5c9a73"
May 26 17:06:56.738776 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:56.738716    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-mvfcs" podUID="b5935fea-b672-471d-a707-a16fc6f3203e"
May 26 17:06:56.775967 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:56.775822    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"]
May 26 17:06:56.849790 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:06:56.849697    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lztjw" podUID="a13dbe45-e72d-4315-a2a1-c4736202fd9b"
May 26 17:06:57.080651 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:06:57.080620    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380972e0_8864_4ad5_8e16_85ce0e8a34b2.slice/crio-61eceaa1f45e79fc1d22a244f324c2e7ad791dc90cf77fadb209cf242d9f64e9 WatchSource:0}: Error finding container 61eceaa1f45e79fc1d22a244f324c2e7ad791dc90cf77fadb209cf242d9f64e9: Status 404 returned error can't find the container with id 61eceaa1f45e79fc1d22a244f324c2e7ad791dc90cf77fadb209cf242d9f64e9
May 26 17:06:57.357715 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:57.357627    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f" event={"ID":"e77b3862-aab3-485a-8910-bfa16b5c9050","Type":"ContainerStarted","Data":"57b71d0815b6753edd0034b3644810590caf57f05b0c156a7f58e60693430a6b"}
May 26 17:06:57.357715 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:57.357666    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f" event={"ID":"e77b3862-aab3-485a-8910-bfa16b5c9050","Type":"ContainerStarted","Data":"b24bc7433c15a233699277237da791c2bdec495b83a17556966ff4cf69fe6b76"}
May 26 17:06:57.357715 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:57.357679    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f" event={"ID":"e77b3862-aab3-485a-8910-bfa16b5c9050","Type":"ContainerStarted","Data":"7a9a55e2c6e582ad767e089d527380d2058a9c71ade0e23452b70a56d5a3789f"}
May 26 17:06:57.357974 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:57.357878    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:06:57.359093 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:57.359066    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx" event={"ID":"8ef69f78-f149-4702-b560-d1a103725d7b","Type":"ContainerStarted","Data":"fc6022e4e777fa800d28bcf543fc39a31567d6aa19288ee5fe844536df06114e"}
May 26 17:06:57.360163 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:57.360143    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx" event={"ID":"380972e0-8864-4ad5-8e16-85ce0e8a34b2","Type":"ContainerStarted","Data":"61eceaa1f45e79fc1d22a244f324c2e7ad791dc90cf77fadb209cf242d9f64e9"}
May 26 17:06:57.360163 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:57.360155    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4c8fb"
May 26 17:06:57.381110 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:57.381050    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f" podStartSLOduration=2.079219014 podStartE2EDuration="5.381001415s" podCreationTimestamp="2026-05-26 17:06:52 +0000 UTC" firstStartedPulling="2026-05-26 17:06:53.348889716 +0000 UTC m=+154.073979209" lastFinishedPulling="2026-05-26 17:06:56.650672127 +0000 UTC m=+157.375761610" observedRunningTime="2026-05-26 17:06:57.380783825 +0000 UTC m=+158.105873327" watchObservedRunningTime="2026-05-26 17:06:57.381001415 +0000 UTC m=+158.106090917"
May 26 17:06:57.398891 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:57.398844    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx" podStartSLOduration=1.03558481 podStartE2EDuration="2.398832077s" podCreationTimestamp="2026-05-26 17:06:55 +0000 UTC" firstStartedPulling="2026-05-26 17:06:55.745489341 +0000 UTC m=+156.470578819" lastFinishedPulling="2026-05-26 17:06:57.108736607 +0000 UTC m=+157.833826086" observedRunningTime="2026-05-26 17:06:57.397591791 +0000 UTC m=+158.122681292" watchObservedRunningTime="2026-05-26 17:06:57.398832077 +0000 UTC m=+158.123921577"
May 26 17:06:59.368026 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:59.367970    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx" event={"ID":"380972e0-8864-4ad5-8e16-85ce0e8a34b2","Type":"ContainerStarted","Data":"97e3ee3904b75e720c35197c746de38e15ea01659f83a50baae55951ae42eeca"}
May 26 17:06:59.368418 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:59.368208    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"
May 26 17:06:59.373089 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:59.373065    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx"
May 26 17:06:59.383502 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:06:59.383458    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-d5zwx" podStartSLOduration=3.039033297 podStartE2EDuration="4.383444345s" podCreationTimestamp="2026-05-26 17:06:55 +0000 UTC" firstStartedPulling="2026-05-26 17:06:57.082663977 +0000 UTC m=+157.807753456" lastFinishedPulling="2026-05-26 17:06:58.427075014 +0000 UTC m=+159.152164504" observedRunningTime="2026-05-26 17:06:59.38282401 +0000 UTC m=+160.107913513" watchObservedRunningTime="2026-05-26 17:06:59.383444345 +0000 UTC m=+160.108533845"
May 26 17:07:01.654871 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:01.654816    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:07:01.655303 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:01.654959    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:07:01.657394 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:01.657368    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5935fea-b672-471d-a707-a16fc6f3203e-cert\") pod \"ingress-canary-mvfcs\" (UID: \"b5935fea-b672-471d-a707-a16fc6f3203e\") " pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:07:01.657497 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:01.657392    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91c3c9fd-a70c-4e23-85db-d5544f5c9a73-metrics-tls\") pod \"dns-default-4c8fb\" (UID: \"91c3c9fd-a70c-4e23-85db-d5544f5c9a73\") " pod="openshift-dns/dns-default-4c8fb"
May 26 17:07:01.864099 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:01.864069    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pnxz5\""
May 26 17:07:01.871636 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:01.871613    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4c8fb"
May 26 17:07:01.998640 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:01.998615    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4c8fb"]
May 26 17:07:02.000603 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:07:02.000574    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c3c9fd_a70c_4e23_85db_d5544f5c9a73.slice/crio-542b453d9a62482ddc97b4886ce181b08728d57494e221b8df7adeed5bfc3058 WatchSource:0}: Error finding container 542b453d9a62482ddc97b4886ce181b08728d57494e221b8df7adeed5bfc3058: Status 404 returned error can't find the container with id 542b453d9a62482ddc97b4886ce181b08728d57494e221b8df7adeed5bfc3058
May 26 17:07:02.379480 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:02.379444    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4c8fb" event={"ID":"91c3c9fd-a70c-4e23-85db-d5544f5c9a73","Type":"ContainerStarted","Data":"542b453d9a62482ddc97b4886ce181b08728d57494e221b8df7adeed5bfc3058"}
May 26 17:07:03.371989 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:03.371961    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7dc4d8cdc8-p7m7f"
May 26 17:07:04.392203 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:04.392170    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4c8fb" event={"ID":"91c3c9fd-a70c-4e23-85db-d5544f5c9a73","Type":"ContainerStarted","Data":"5eb567a34e49cb32d48a5bcf3dc32b998ecb4946f6d98ca8fd60376546e46e76"}
May 26 17:07:04.392203 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:04.392207    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4c8fb" event={"ID":"91c3c9fd-a70c-4e23-85db-d5544f5c9a73","Type":"ContainerStarted","Data":"c5cb2ba73e641aa4f9d6ada658a63b36d2d3af060e0aebb406cec8550a7dd3be"}
May 26 17:07:04.392678 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:04.392437    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4c8fb"
May 26 17:07:04.412739 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:04.412687    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-4c8fb" podStartSLOduration=129.906140944 podStartE2EDuration="2m11.412670983s" podCreationTimestamp="2026-05-26 17:04:53 +0000 UTC" firstStartedPulling="2026-05-26 17:07:02.002513026 +0000 UTC m=+162.727602505" lastFinishedPulling="2026-05-26 17:07:03.509043062 +0000 UTC m=+164.234132544" observedRunningTime="2026-05-26 17:07:04.41182763 +0000 UTC m=+165.136917132" watchObservedRunningTime="2026-05-26 17:07:04.412670983 +0000 UTC m=+165.137760512"
May 26 17:07:06.411667 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.411632    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-dvsmd"]
May 26 17:07:06.413937 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.413919    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-dvsmd"
May 26 17:07:06.416753 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.416724    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ckhhn\""
May 26 17:07:06.416907 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.416773    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 26 17:07:06.416907 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.416790    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 26 17:07:06.422923 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.422899    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-dvsmd"]
May 26 17:07:06.499791 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.499754    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzj9\" (UniqueName: \"kubernetes.io/projected/27ace5ae-76be-4c79-93ce-e4095cb098ab-kube-api-access-5wzj9\") pod \"downloads-57bd96b888-dvsmd\" (UID: \"27ace5ae-76be-4c79-93ce-e4095cb098ab\") " pod="openshift-console/downloads-57bd96b888-dvsmd"
May 26 17:07:06.600921 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.600883    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzj9\" (UniqueName: \"kubernetes.io/projected/27ace5ae-76be-4c79-93ce-e4095cb098ab-kube-api-access-5wzj9\") pod \"downloads-57bd96b888-dvsmd\" (UID: \"27ace5ae-76be-4c79-93ce-e4095cb098ab\") " pod="openshift-console/downloads-57bd96b888-dvsmd"
May 26 17:07:06.609673 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.609648    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzj9\" (UniqueName: \"kubernetes.io/projected/27ace5ae-76be-4c79-93ce-e4095cb098ab-kube-api-access-5wzj9\") pod \"downloads-57bd96b888-dvsmd\" (UID: \"27ace5ae-76be-4c79-93ce-e4095cb098ab\") " pod="openshift-console/downloads-57bd96b888-dvsmd"
May 26 17:07:06.723710 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.723615    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-dvsmd"
May 26 17:07:06.853179 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:06.853153    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-dvsmd"]
May 26 17:07:06.855584 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:07:06.855556    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ace5ae_76be_4c79_93ce_e4095cb098ab.slice/crio-fe79ed2cdfb9125b9772afaf2001a0b1065381a575878135057f0da6712b6ee9 WatchSource:0}: Error finding container fe79ed2cdfb9125b9772afaf2001a0b1065381a575878135057f0da6712b6ee9: Status 404 returned error can't find the container with id fe79ed2cdfb9125b9772afaf2001a0b1065381a575878135057f0da6712b6ee9
May 26 17:07:07.401525 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:07.401485    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-dvsmd" event={"ID":"27ace5ae-76be-4c79-93ce-e4095cb098ab","Type":"ContainerStarted","Data":"fe79ed2cdfb9125b9772afaf2001a0b1065381a575878135057f0da6712b6ee9"}
May 26 17:07:09.838646 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:09.838573    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:07:09.839105 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:09.838573    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:07:09.841862 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:09.841841    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jvpqs\""
May 26 17:07:09.849336 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:09.849318    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mvfcs"
May 26 17:07:09.977965 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:09.977927    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mvfcs"]
May 26 17:07:09.980772 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:07:09.980744    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5935fea_b672_471d_a707_a16fc6f3203e.slice/crio-75d5d2881315c9bbca493f7642bb9a05f69d55da73af5b0c5e020425f5505b84 WatchSource:0}: Error finding container 75d5d2881315c9bbca493f7642bb9a05f69d55da73af5b0c5e020425f5505b84: Status 404 returned error can't find the container with id 75d5d2881315c9bbca493f7642bb9a05f69d55da73af5b0c5e020425f5505b84
May 26 17:07:10.410681 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:10.410643    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mvfcs" event={"ID":"b5935fea-b672-471d-a707-a16fc6f3203e","Type":"ContainerStarted","Data":"75d5d2881315c9bbca493f7642bb9a05f69d55da73af5b0c5e020425f5505b84"}
May 26 17:07:12.419825 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:12.419741    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mvfcs" event={"ID":"b5935fea-b672-471d-a707-a16fc6f3203e","Type":"ContainerStarted","Data":"1c9e063258c2659a089b48c4b2c8262589e67a301cd6ac1331f43d7a94e2ab43"}
May 26 17:07:12.435718 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:12.435667    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mvfcs" podStartSLOduration=137.300091572 podStartE2EDuration="2m19.435651463s" podCreationTimestamp="2026-05-26 17:04:53 +0000 UTC" firstStartedPulling="2026-05-26 17:07:09.982649855 +0000 UTC m=+170.707739337" lastFinishedPulling="2026-05-26 17:07:12.118209746 +0000 UTC m=+172.843299228" observedRunningTime="2026-05-26 17:07:12.435649089 +0000 UTC m=+173.160738591" watchObservedRunningTime="2026-05-26 17:07:12.435651463 +0000 UTC m=+173.160740964"
May 26 17:07:14.397546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:14.397511    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4c8fb"
May 26 17:07:15.513618 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.513412    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bcf6cbd6c-tgdwl"]
May 26 17:07:15.518277 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.518248    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.521196 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.521172    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 26 17:07:15.522287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.522258    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 26 17:07:15.522418 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.522300    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dprqv\""
May 26 17:07:15.522548 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.522524    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 26 17:07:15.522548 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.522543    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 26 17:07:15.522817 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.522593    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 26 17:07:15.527429 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.527407    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcf6cbd6c-tgdwl"]
May 26 17:07:15.594055 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.594021    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:07:15.594055 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.594063    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:07:15.688256 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.688210    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqr64\" (UniqueName: \"kubernetes.io/projected/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-kube-api-access-wqr64\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.688451 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.688285    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-oauth-serving-cert\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.688451 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.688337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-service-ca\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.688560 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.688549    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-oauth-config\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.688633 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.688616    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-serving-cert\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.688684 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.688649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-config\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.789481 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.789384    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-config\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.789653 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.789489    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wqr64\" (UniqueName: \"kubernetes.io/projected/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-kube-api-access-wqr64\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.789653 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.789527    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-oauth-serving-cert\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.789653 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.789557    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-service-ca\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.789816 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.789675    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-oauth-config\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.789816 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.789728    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-serving-cert\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.790134 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.790110    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-config\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.790242 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.790215    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-oauth-serving-cert\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.790302 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.790287    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-service-ca\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.792185 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.792162    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-oauth-config\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.792290 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.792195    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-serving-cert\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.797607 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.797587    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqr64\" (UniqueName: \"kubernetes.io/projected/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-kube-api-access-wqr64\") pod \"console-5bcf6cbd6c-tgdwl\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") " pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.830227 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.830200    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:15.966761 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:15.966648    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcf6cbd6c-tgdwl"]
May 26 17:07:15.969536 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:07:15.969499    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff30fa3_476a_4c66_9eaf_6f416c1cbae9.slice/crio-c541b7b3be4ffd94e2fd518d44c86695932febd921109d25117d5dc258bcbbcc WatchSource:0}: Error finding container c541b7b3be4ffd94e2fd518d44c86695932febd921109d25117d5dc258bcbbcc: Status 404 returned error can't find the container with id c541b7b3be4ffd94e2fd518d44c86695932febd921109d25117d5dc258bcbbcc
May 26 17:07:16.438087 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:16.438046    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcf6cbd6c-tgdwl" event={"ID":"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9","Type":"ContainerStarted","Data":"c541b7b3be4ffd94e2fd518d44c86695932febd921109d25117d5dc258bcbbcc"}
May 26 17:07:17.443692 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:17.443648    2571 generic.go:358] "Generic (PLEG): container finished" podID="5f96e4a7-3baa-47a8-b688-ed682aaeba85" containerID="341919872b75a990d34f85e111bc4be7a7f05687aa099603851ffe866177fdf7" exitCode=0
May 26 17:07:17.444272 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:17.443724    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-w47qs" event={"ID":"5f96e4a7-3baa-47a8-b688-ed682aaeba85","Type":"ContainerDied","Data":"341919872b75a990d34f85e111bc4be7a7f05687aa099603851ffe866177fdf7"}
May 26 17:07:17.444272 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:17.444111    2571 scope.go:117] "RemoveContainer" containerID="341919872b75a990d34f85e111bc4be7a7f05687aa099603851ffe866177fdf7"
May 26 17:07:18.449438 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:18.449400    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-w47qs" event={"ID":"5f96e4a7-3baa-47a8-b688-ed682aaeba85","Type":"ContainerStarted","Data":"e80608b931e5fd4af10a4895027bd3495d4b3fab56338ddc4e4b6a4ea541b23f"}
May 26 17:07:20.458208 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:20.458167    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcf6cbd6c-tgdwl" event={"ID":"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9","Type":"ContainerStarted","Data":"64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920"}
May 26 17:07:20.476104 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:20.476046    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5bcf6cbd6c-tgdwl" podStartSLOduration=2.010188237 podStartE2EDuration="5.476004162s" podCreationTimestamp="2026-05-26 17:07:15 +0000 UTC" firstStartedPulling="2026-05-26 17:07:15.971839212 +0000 UTC m=+176.696928692" lastFinishedPulling="2026-05-26 17:07:19.437655125 +0000 UTC m=+180.162744617" observedRunningTime="2026-05-26 17:07:20.474711171 +0000 UTC m=+181.199800686" watchObservedRunningTime="2026-05-26 17:07:20.476004162 +0000 UTC m=+181.201093662"
May 26 17:07:25.835584 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:25.835552    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:25.836006 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:25.835596    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:25.836345 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:25.836324    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:26.488444 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:26.488416    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:27.490029 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:27.489964    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-dvsmd" event={"ID":"27ace5ae-76be-4c79-93ce-e4095cb098ab","Type":"ContainerStarted","Data":"a13af9d9284954214fb6816ba0eda8af828b47aac152d4a59d0dab203b49e6e2"}
May 26 17:07:27.515957 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:27.515900    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-dvsmd" podStartSLOduration=1.879094805 podStartE2EDuration="21.515886398s" podCreationTimestamp="2026-05-26 17:07:06 +0000 UTC" firstStartedPulling="2026-05-26 17:07:06.857626432 +0000 UTC m=+167.582715911" lastFinishedPulling="2026-05-26 17:07:26.494418025 +0000 UTC m=+187.219507504" observedRunningTime="2026-05-26 17:07:27.514099716 +0000 UTC m=+188.239189218" watchObservedRunningTime="2026-05-26 17:07:27.515886398 +0000 UTC m=+188.240975899"
May 26 17:07:28.493651 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:28.493617    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-dvsmd"
May 26 17:07:28.508505 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:28.508476    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-dvsmd"
May 26 17:07:29.498513 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:29.498475    2571 generic.go:358] "Generic (PLEG): container finished" podID="a754be44-8031-4393-ae82-46994bc250ae" containerID="1a905578a2064b3034cee4a641912bafe58ea3c19454724e0a8a211979de055d" exitCode=0
May 26 17:07:29.498966 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:29.498557    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh" event={"ID":"a754be44-8031-4393-ae82-46994bc250ae","Type":"ContainerDied","Data":"1a905578a2064b3034cee4a641912bafe58ea3c19454724e0a8a211979de055d"}
May 26 17:07:29.499111 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:29.499094    2571 scope.go:117] "RemoveContainer" containerID="1a905578a2064b3034cee4a641912bafe58ea3c19454724e0a8a211979de055d"
May 26 17:07:30.503326 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:30.503282    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-kkdjh" event={"ID":"a754be44-8031-4393-ae82-46994bc250ae","Type":"ContainerStarted","Data":"df8143035f5b8dd226c84ee929601b0f6e4a42b5ffb3692f8a3fc3c7112a5111"}
May 26 17:07:34.497804 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:34.497765    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcf6cbd6c-tgdwl"]
May 26 17:07:35.599469 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:35.599433    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:07:35.603362 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:35.603336    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d8d68d57b-m8qpx"
May 26 17:07:41.541548 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:41.541516    2571 generic.go:358] "Generic (PLEG): container finished" podID="df3cb020-a879-45c7-8fff-b38a4b5ccf30" containerID="e9e7b570e527504313a069639d29d33fe861fe860a97910855a1e3d528b45e6e" exitCode=0
May 26 17:07:41.541956 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:41.541561    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m" event={"ID":"df3cb020-a879-45c7-8fff-b38a4b5ccf30","Type":"ContainerDied","Data":"e9e7b570e527504313a069639d29d33fe861fe860a97910855a1e3d528b45e6e"}
May 26 17:07:41.541956 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:41.541901    2571 scope.go:117] "RemoveContainer" containerID="e9e7b570e527504313a069639d29d33fe861fe860a97910855a1e3d528b45e6e"
May 26 17:07:42.545657 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:42.545620    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-gbm7m" event={"ID":"df3cb020-a879-45c7-8fff-b38a4b5ccf30","Type":"ContainerStarted","Data":"659db0a1300cb8848f4c3d481079d40a15006019e74e653bc772d610ea315d3e"}
May 26 17:07:59.525196 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.520895    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5bcf6cbd6c-tgdwl" podUID="7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" containerName="console" containerID="cri-o://64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920" gracePeriod=15
May 26 17:07:59.794645 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.794622    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bcf6cbd6c-tgdwl_7ff30fa3-476a-4c66-9eaf-6f416c1cbae9/console/0.log"
May 26 17:07:59.794772 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.794710    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:07:59.799239 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.799217    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-oauth-config\") pod \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") "
May 26 17:07:59.799362 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.799300    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-service-ca\") pod \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") "
May 26 17:07:59.799362 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.799335    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqr64\" (UniqueName: \"kubernetes.io/projected/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-kube-api-access-wqr64\") pod \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") "
May 26 17:07:59.799482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.799361    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-config\") pod \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") "
May 26 17:07:59.799537 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.799501    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-oauth-serving-cert\") pod \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") "
May 26 17:07:59.799597 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.799550    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-serving-cert\") pod \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\" (UID: \"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9\") "
May 26 17:07:59.799856 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.799826    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-service-ca" (OuterVolumeSpecName: "service-ca") pod "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" (UID: "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:07:59.799856 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.799844    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-config" (OuterVolumeSpecName: "console-config") pod "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" (UID: "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:07:59.800060 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.799947    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" (UID: "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:07:59.801602 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.801574    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-kube-api-access-wqr64" (OuterVolumeSpecName: "kube-api-access-wqr64") pod "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" (UID: "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9"). InnerVolumeSpecName "kube-api-access-wqr64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:07:59.801698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.801580    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" (UID: "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:07:59.801698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.801582    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" (UID: "7ff30fa3-476a-4c66-9eaf-6f416c1cbae9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:07:59.900704 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.900671    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-service-ca\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:07:59.900704 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.900700    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqr64\" (UniqueName: \"kubernetes.io/projected/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-kube-api-access-wqr64\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:07:59.900704 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.900713    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-config\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:07:59.900933 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.900722    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-oauth-serving-cert\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:07:59.900933 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.900731    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-serving-cert\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:07:59.900933 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:07:59.900739    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9-console-oauth-config\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:08:00.595927 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.595897    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bcf6cbd6c-tgdwl_7ff30fa3-476a-4c66-9eaf-6f416c1cbae9/console/0.log"
May 26 17:08:00.596401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.595938    2571 generic.go:358] "Generic (PLEG): container finished" podID="7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" containerID="64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920" exitCode=2
May 26 17:08:00.596401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.595971    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcf6cbd6c-tgdwl" event={"ID":"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9","Type":"ContainerDied","Data":"64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920"}
May 26 17:08:00.596401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.596026    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcf6cbd6c-tgdwl"
May 26 17:08:00.596401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.596042    2571 scope.go:117] "RemoveContainer" containerID="64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920"
May 26 17:08:00.596401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.596029    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcf6cbd6c-tgdwl" event={"ID":"7ff30fa3-476a-4c66-9eaf-6f416c1cbae9","Type":"ContainerDied","Data":"c541b7b3be4ffd94e2fd518d44c86695932febd921109d25117d5dc258bcbbcc"}
May 26 17:08:00.609046 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.609024    2571 scope.go:117] "RemoveContainer" containerID="64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920"
May 26 17:08:00.609315 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:08:00.609294    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920\": container with ID starting with 64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920 not found: ID does not exist" containerID="64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920"
May 26 17:08:00.609358 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.609324    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920"} err="failed to get container status \"64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920\": rpc error: code = NotFound desc = could not find container \"64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920\": container with ID starting with 64f76fec29dcd773cb273f5011d5ca165de52de1cb32909cee58423c4c506920 not found: ID does not exist"
May 26 17:08:00.613892 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.613870    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcf6cbd6c-tgdwl"]
May 26 17:08:00.619253 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:00.619233    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bcf6cbd6c-tgdwl"]
May 26 17:08:01.834773 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:01.834742    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" path="/var/lib/kubelet/pods/7ff30fa3-476a-4c66-9eaf-6f416c1cbae9/volumes"
May 26 17:08:15.117856 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.117818    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bd9755968-2d2zx"]
May 26 17:08:15.118441 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.118183    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" containerName="console"
May 26 17:08:15.118441 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.118195    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" containerName="console"
May 26 17:08:15.118441 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.118267    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="7ff30fa3-476a-4c66-9eaf-6f416c1cbae9" containerName="console"
May 26 17:08:15.120341 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.120324    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.123328 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.123308    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 26 17:08:15.123468 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.123373    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t8wws\""
May 26 17:08:15.123656 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.123636    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 26 17:08:15.123741 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.123686    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 26 17:08:15.123798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.123782    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 26 17:08:15.125146 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.125130    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 26 17:08:15.128666 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.128647    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 26 17:08:15.133687 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.133661    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bd9755968-2d2zx"]
May 26 17:08:15.232877 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.232839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381a615-e822-4c85-89cc-fd790ef6a0da-serving-certs-ca-bundle\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.232877 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.232875    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5d7n\" (UniqueName: \"kubernetes.io/projected/6381a615-e822-4c85-89cc-fd790ef6a0da-kube-api-access-v5d7n\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.233117 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.232900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381a615-e822-4c85-89cc-fd790ef6a0da-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.233117 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.232960    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-federate-client-tls\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.233117 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.233048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-telemeter-client-tls\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.233117 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.233074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.233244 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.233158    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-secret-telemeter-client\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.233244 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.233179    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6381a615-e822-4c85-89cc-fd790ef6a0da-metrics-client-ca\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.334306 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.334274    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-secret-telemeter-client\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.334455 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.334310    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6381a615-e822-4c85-89cc-fd790ef6a0da-metrics-client-ca\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.334455 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.334338    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381a615-e822-4c85-89cc-fd790ef6a0da-serving-certs-ca-bundle\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.334455 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.334364    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v5d7n\" (UniqueName: \"kubernetes.io/projected/6381a615-e822-4c85-89cc-fd790ef6a0da-kube-api-access-v5d7n\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.334570 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.334535    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381a615-e822-4c85-89cc-fd790ef6a0da-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.334619 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.334588    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-federate-client-tls\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.334679 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.334642    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-telemeter-client-tls\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.334679 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.334671    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.335126 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.335075    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381a615-e822-4c85-89cc-fd790ef6a0da-serving-certs-ca-bundle\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.335265 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.335208    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6381a615-e822-4c85-89cc-fd790ef6a0da-metrics-client-ca\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.335618 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.335595    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381a615-e822-4c85-89cc-fd790ef6a0da-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.337057 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.337007    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-federate-client-tls\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.337154 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.337137    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.337230 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.337212    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-secret-telemeter-client\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.337269 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.337251    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6381a615-e822-4c85-89cc-fd790ef6a0da-telemeter-client-tls\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.341842 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.341821    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5d7n\" (UniqueName: \"kubernetes.io/projected/6381a615-e822-4c85-89cc-fd790ef6a0da-kube-api-access-v5d7n\") pod \"telemeter-client-5bd9755968-2d2zx\" (UID: \"6381a615-e822-4c85-89cc-fd790ef6a0da\") " pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.430175 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.430083    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx"
May 26 17:08:15.560973 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.560944    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bd9755968-2d2zx"]
May 26 17:08:15.562387 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:08:15.562357    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6381a615_e822_4c85_89cc_fd790ef6a0da.slice/crio-ccb803406ca43e30671b3aa3578919620d308897072601e8e2f31dd52e4d553c WatchSource:0}: Error finding container ccb803406ca43e30671b3aa3578919620d308897072601e8e2f31dd52e4d553c: Status 404 returned error can't find the container with id ccb803406ca43e30671b3aa3578919620d308897072601e8e2f31dd52e4d553c
May 26 17:08:15.645597 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:15.645555    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx" event={"ID":"6381a615-e822-4c85-89cc-fd790ef6a0da","Type":"ContainerStarted","Data":"ccb803406ca43e30671b3aa3578919620d308897072601e8e2f31dd52e4d553c"}
May 26 17:08:17.653243 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:17.653209    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx" event={"ID":"6381a615-e822-4c85-89cc-fd790ef6a0da","Type":"ContainerStarted","Data":"4482747e187336988c9b02331896c6854f0b09ab0a7fe7dc0e126a68c4f8191c"}
May 26 17:08:19.661330 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:19.661298    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx" event={"ID":"6381a615-e822-4c85-89cc-fd790ef6a0da","Type":"ContainerStarted","Data":"31b5aa69ad408c434cbd52e42657680c977187fed45a85b9710c7630a140af92"}
May 26 17:08:19.661330 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:19.661332    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx" event={"ID":"6381a615-e822-4c85-89cc-fd790ef6a0da","Type":"ContainerStarted","Data":"4f05c5f69878f9d3d1641de7ba9ff6118e288abee05e3b1f23c031405bfb753d"}
May 26 17:08:19.689446 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:19.689397    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bd9755968-2d2zx" podStartSLOduration=1.559107725 podStartE2EDuration="4.689380718s" podCreationTimestamp="2026-05-26 17:08:15 +0000 UTC" firstStartedPulling="2026-05-26 17:08:15.564195073 +0000 UTC m=+236.289284555" lastFinishedPulling="2026-05-26 17:08:18.69446807 +0000 UTC m=+239.419557548" observedRunningTime="2026-05-26 17:08:19.68734153 +0000 UTC m=+240.412431031" watchObservedRunningTime="2026-05-26 17:08:19.689380718 +0000 UTC m=+240.414470226"
May 26 17:08:20.360142 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.360112    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5cdc748d-zmtg2"]
May 26 17:08:20.362979 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.362958    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.367235 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.367211    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 26 17:08:20.367367 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.367328    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 26 17:08:20.367367 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.367339    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 26 17:08:20.367367 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.367345    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 26 17:08:20.367523 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.367410    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 26 17:08:20.367800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.367781    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dprqv\""
May 26 17:08:20.371852 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.371824    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 26 17:08:20.375621 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.375600    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5cdc748d-zmtg2"]
May 26 17:08:20.483949 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.483914    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxj8\" (UniqueName: \"kubernetes.io/projected/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-kube-api-access-6bxj8\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.484152 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.483972    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-config\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.484152 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.484066    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-oauth-config\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.484247 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.484164    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-serving-cert\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.484247 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.484192    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-trusted-ca-bundle\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.484247 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.484233    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-oauth-serving-cert\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.484363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.484258    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-service-ca\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.585372 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.585338    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-serving-cert\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.585372 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.585377    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-trusted-ca-bundle\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.585745 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.585399    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-oauth-serving-cert\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.585745 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.585426    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-service-ca\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.585745 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.585471    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxj8\" (UniqueName: \"kubernetes.io/projected/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-kube-api-access-6bxj8\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.585745 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.585509    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-config\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.585745 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.585548    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-oauth-config\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.586255 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.586232    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-oauth-serving-cert\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.586381 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.586324    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-config\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.586443 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.586427    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-service-ca\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.586495 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.586447    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-trusted-ca-bundle\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.588045 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.588005    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-oauth-config\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.588150 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.588134    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-serving-cert\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.593371 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.593350    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxj8\" (UniqueName: \"kubernetes.io/projected/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-kube-api-access-6bxj8\") pod \"console-b5cdc748d-zmtg2\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") " pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.673554 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.673477    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:20.801911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:20.801524    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5cdc748d-zmtg2"]
May 26 17:08:20.804392 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:08:20.804360    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ca74f3_30b9_49bd_b4bf_95fbde45afc6.slice/crio-b07ba094d0474238d461f57013906c2a344c6aaef8a4d03e073af294e865a690 WatchSource:0}: Error finding container b07ba094d0474238d461f57013906c2a344c6aaef8a4d03e073af294e865a690: Status 404 returned error can't find the container with id b07ba094d0474238d461f57013906c2a344c6aaef8a4d03e073af294e865a690
May 26 17:08:21.668262 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:21.668232    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5cdc748d-zmtg2" event={"ID":"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6","Type":"ContainerStarted","Data":"daaf01662aaa9868a8782fdc51cb6beb7dc589c4fbddd0914d7d644257596036"}
May 26 17:08:21.668262 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:21.668263    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5cdc748d-zmtg2" event={"ID":"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6","Type":"ContainerStarted","Data":"b07ba094d0474238d461f57013906c2a344c6aaef8a4d03e073af294e865a690"}
May 26 17:08:21.686443 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:21.686396    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-b5cdc748d-zmtg2" podStartSLOduration=1.686381816 podStartE2EDuration="1.686381816s" podCreationTimestamp="2026-05-26 17:08:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:08:21.685465396 +0000 UTC m=+242.410554886" watchObservedRunningTime="2026-05-26 17:08:21.686381816 +0000 UTC m=+242.411471316"
May 26 17:08:30.674520 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:30.674475    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:30.675042 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:30.674571    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:30.679091 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:30.679072    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:30.697083 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:30.697059    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:08:31.585528 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:31.585481    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:08:31.587816 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:31.587787    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13dbe45-e72d-4315-a2a1-c4736202fd9b-metrics-certs\") pod \"network-metrics-daemon-lztjw\" (UID: \"a13dbe45-e72d-4315-a2a1-c4736202fd9b\") " pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:08:31.742252 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:31.742219    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brj54\""
May 26 17:08:31.749953 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:31.749931    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lztjw"
May 26 17:08:31.879106 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:31.878961    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lztjw"]
May 26 17:08:31.881645 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:08:31.881613    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13dbe45_e72d_4315_a2a1_c4736202fd9b.slice/crio-ee2d3266166083fd5a8283f56e241c9c3d02dd7bfb0b2325e08e9cc521b3decf WatchSource:0}: Error finding container ee2d3266166083fd5a8283f56e241c9c3d02dd7bfb0b2325e08e9cc521b3decf: Status 404 returned error can't find the container with id ee2d3266166083fd5a8283f56e241c9c3d02dd7bfb0b2325e08e9cc521b3decf
May 26 17:08:32.700794 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:32.700752    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lztjw" event={"ID":"a13dbe45-e72d-4315-a2a1-c4736202fd9b","Type":"ContainerStarted","Data":"ee2d3266166083fd5a8283f56e241c9c3d02dd7bfb0b2325e08e9cc521b3decf"}
May 26 17:08:33.705931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:33.705880    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lztjw" event={"ID":"a13dbe45-e72d-4315-a2a1-c4736202fd9b","Type":"ContainerStarted","Data":"5fcd99679193710e0cb4909878ef6adb00ee41b243700f4c2ddc8b1b18d60ce4"}
May 26 17:08:33.705931 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:33.705933    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lztjw" event={"ID":"a13dbe45-e72d-4315-a2a1-c4736202fd9b","Type":"ContainerStarted","Data":"761ebcf1dff386f3905ef7acf02a956376777669b689a55b80245811d087cbd0"}
May 26 17:08:33.724207 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:08:33.724162    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lztjw" podStartSLOduration=253.732742923 podStartE2EDuration="4m14.72415037s" podCreationTimestamp="2026-05-26 17:04:19 +0000 UTC" firstStartedPulling="2026-05-26 17:08:31.883593809 +0000 UTC m=+252.608683291" lastFinishedPulling="2026-05-26 17:08:32.875001249 +0000 UTC m=+253.600090738" observedRunningTime="2026-05-26 17:08:33.722529695 +0000 UTC m=+254.447619196" watchObservedRunningTime="2026-05-26 17:08:33.72415037 +0000 UTC m=+254.449239868"
May 26 17:09:19.727640 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:09:19.727608    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:09:19.728262 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:09:19.727849    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:09:19.737876 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:09:19.737853    2571 kubelet.go:1628] "Image garbage collection succeeded"
May 26 17:09:36.798921 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:09:36.798884    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5cdc748d-zmtg2"]
May 26 17:10:01.818875 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:01.818815    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-b5cdc748d-zmtg2" podUID="d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" containerName="console" containerID="cri-o://daaf01662aaa9868a8782fdc51cb6beb7dc589c4fbddd0914d7d644257596036" gracePeriod=15
May 26 17:10:01.976577 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:01.976551    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b5cdc748d-zmtg2_d7ca74f3-30b9-49bd-b4bf-95fbde45afc6/console/0.log"
May 26 17:10:01.976722 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:01.976586    2571 generic.go:358] "Generic (PLEG): container finished" podID="d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" containerID="daaf01662aaa9868a8782fdc51cb6beb7dc589c4fbddd0914d7d644257596036" exitCode=2
May 26 17:10:01.976722 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:01.976675    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5cdc748d-zmtg2" event={"ID":"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6","Type":"ContainerDied","Data":"daaf01662aaa9868a8782fdc51cb6beb7dc589c4fbddd0914d7d644257596036"}
May 26 17:10:02.062871 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.062848    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b5cdc748d-zmtg2_d7ca74f3-30b9-49bd-b4bf-95fbde45afc6/console/0.log"
May 26 17:10:02.062995 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.062907    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:10:02.114113 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114005    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-serving-cert\") pod \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") "
May 26 17:10:02.114113 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114083    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bxj8\" (UniqueName: \"kubernetes.io/projected/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-kube-api-access-6bxj8\") pod \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") "
May 26 17:10:02.114113 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114099    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-oauth-config\") pod \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") "
May 26 17:10:02.114359 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114128    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-service-ca\") pod \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") "
May 26 17:10:02.114359 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114155    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-trusted-ca-bundle\") pod \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") "
May 26 17:10:02.114359 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114217    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-config\") pod \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") "
May 26 17:10:02.114359 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114251    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-oauth-serving-cert\") pod \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\" (UID: \"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6\") "
May 26 17:10:02.114615 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114589    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-service-ca" (OuterVolumeSpecName: "service-ca") pod "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" (UID: "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:10:02.114745 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114633    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-config" (OuterVolumeSpecName: "console-config") pod "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" (UID: "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:10:02.114745 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114713    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" (UID: "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:10:02.114925 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.114756    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" (UID: "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:10:02.116313 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.116292    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" (UID: "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:10:02.116580 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.116561    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-kube-api-access-6bxj8" (OuterVolumeSpecName: "kube-api-access-6bxj8") pod "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" (UID: "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6"). InnerVolumeSpecName "kube-api-access-6bxj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:10:02.116580 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.116559    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" (UID: "d7ca74f3-30b9-49bd-b4bf-95fbde45afc6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:10:02.215284 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.215249    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-config\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:10:02.215284 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.215278    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-oauth-serving-cert\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:10:02.215284 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.215288    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-serving-cert\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:10:02.215514 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.215297    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxj8\" (UniqueName: \"kubernetes.io/projected/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-kube-api-access-6bxj8\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:10:02.215514 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.215306    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-console-oauth-config\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:10:02.215514 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.215315    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-service-ca\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:10:02.215514 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.215323    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6-trusted-ca-bundle\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:10:02.981276 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.981246    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b5cdc748d-zmtg2_d7ca74f3-30b9-49bd-b4bf-95fbde45afc6/console/0.log"
May 26 17:10:02.981717 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.981358    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5cdc748d-zmtg2"
May 26 17:10:02.981717 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.981383    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5cdc748d-zmtg2" event={"ID":"d7ca74f3-30b9-49bd-b4bf-95fbde45afc6","Type":"ContainerDied","Data":"b07ba094d0474238d461f57013906c2a344c6aaef8a4d03e073af294e865a690"}
May 26 17:10:02.981717 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:02.981439    2571 scope.go:117] "RemoveContainer" containerID="daaf01662aaa9868a8782fdc51cb6beb7dc589c4fbddd0914d7d644257596036"
May 26 17:10:03.003392 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.003363    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5cdc748d-zmtg2"]
May 26 17:10:03.007171 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.007142    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b5cdc748d-zmtg2"]
May 26 17:10:03.167115 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.167081    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-2xt7z"]
May 26 17:10:03.167505 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.167491    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" containerName="console"
May 26 17:10:03.167550 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.167508    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" containerName="console"
May 26 17:10:03.167584 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.167561    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" containerName="console"
May 26 17:10:03.171616 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.171597    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.174293 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.174274    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 17:10:03.180979 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.180956    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2xt7z"]
May 26 17:10:03.223991 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.223949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e2672b8-f37b-4393-8b5d-2c9dd96bb85f-original-pull-secret\") pod \"global-pull-secret-syncer-2xt7z\" (UID: \"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f\") " pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.223991 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.223990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e2672b8-f37b-4393-8b5d-2c9dd96bb85f-kubelet-config\") pod \"global-pull-secret-syncer-2xt7z\" (UID: \"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f\") " pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.224227 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.224034    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e2672b8-f37b-4393-8b5d-2c9dd96bb85f-dbus\") pod \"global-pull-secret-syncer-2xt7z\" (UID: \"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f\") " pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.325147 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.325113    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e2672b8-f37b-4393-8b5d-2c9dd96bb85f-original-pull-secret\") pod \"global-pull-secret-syncer-2xt7z\" (UID: \"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f\") " pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.325316 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.325158    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e2672b8-f37b-4393-8b5d-2c9dd96bb85f-kubelet-config\") pod \"global-pull-secret-syncer-2xt7z\" (UID: \"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f\") " pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.325316 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.325226    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e2672b8-f37b-4393-8b5d-2c9dd96bb85f-kubelet-config\") pod \"global-pull-secret-syncer-2xt7z\" (UID: \"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f\") " pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.325316 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.325278    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e2672b8-f37b-4393-8b5d-2c9dd96bb85f-dbus\") pod \"global-pull-secret-syncer-2xt7z\" (UID: \"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f\") " pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.325479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.325462    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e2672b8-f37b-4393-8b5d-2c9dd96bb85f-dbus\") pod \"global-pull-secret-syncer-2xt7z\" (UID: \"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f\") " pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.327524 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.327503    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e2672b8-f37b-4393-8b5d-2c9dd96bb85f-original-pull-secret\") pod \"global-pull-secret-syncer-2xt7z\" (UID: \"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f\") " pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.480428 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.480393    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xt7z"
May 26 17:10:03.606813 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.606652    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2xt7z"]
May 26 17:10:03.609359 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:10:03.609328    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2672b8_f37b_4393_8b5d_2c9dd96bb85f.slice/crio-43ac6fc05c1fc19c0b1cf68266c9f589be50874bdd203bd4acdd2df04cfc3362 WatchSource:0}: Error finding container 43ac6fc05c1fc19c0b1cf68266c9f589be50874bdd203bd4acdd2df04cfc3362: Status 404 returned error can't find the container with id 43ac6fc05c1fc19c0b1cf68266c9f589be50874bdd203bd4acdd2df04cfc3362
May 26 17:10:03.610935 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.610918    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 17:10:03.834241 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.834209    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ca74f3-30b9-49bd-b4bf-95fbde45afc6" path="/var/lib/kubelet/pods/d7ca74f3-30b9-49bd-b4bf-95fbde45afc6/volumes"
May 26 17:10:03.987653 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:03.987551    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2xt7z" event={"ID":"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f","Type":"ContainerStarted","Data":"43ac6fc05c1fc19c0b1cf68266c9f589be50874bdd203bd4acdd2df04cfc3362"}
May 26 17:10:09.004383 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:09.004349    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2xt7z" event={"ID":"7e2672b8-f37b-4393-8b5d-2c9dd96bb85f","Type":"ContainerStarted","Data":"6b83e9f42c29d5177a702073ca0de686ab41c4ebfd7e19897244f3649ea9a93c"}
May 26 17:10:09.019837 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:10:09.019789    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2xt7z" podStartSLOduration=1.174497998 podStartE2EDuration="6.019771634s" podCreationTimestamp="2026-05-26 17:10:03 +0000 UTC" firstStartedPulling="2026-05-26 17:10:03.611137556 +0000 UTC m=+344.336227037" lastFinishedPulling="2026-05-26 17:10:08.456411191 +0000 UTC m=+349.181500673" observedRunningTime="2026-05-26 17:10:09.018839847 +0000 UTC m=+349.743929349" watchObservedRunningTime="2026-05-26 17:10:09.019771634 +0000 UTC m=+349.744861135"
May 26 17:11:43.578991 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.578903    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"]
May 26 17:11:43.581291 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.581274    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"
May 26 17:11:43.584064 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.584037    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 26 17:11:43.585333 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.585314    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-bwd75\""
May 26 17:11:43.585436 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.585314    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 26 17:11:43.596503 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.596481    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"]
May 26 17:11:43.766184 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.766135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwf9n\" (UniqueName: \"kubernetes.io/projected/2a13712f-2bef-4ee2-bc14-21157355d517-kube-api-access-lwf9n\") pod \"feast-operator-controller-manager-6dc7cc57cf-l4pwg\" (UID: \"2a13712f-2bef-4ee2-bc14-21157355d517\") " pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"
May 26 17:11:43.866869 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.866779    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lwf9n\" (UniqueName: \"kubernetes.io/projected/2a13712f-2bef-4ee2-bc14-21157355d517-kube-api-access-lwf9n\") pod \"feast-operator-controller-manager-6dc7cc57cf-l4pwg\" (UID: \"2a13712f-2bef-4ee2-bc14-21157355d517\") " pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"
May 26 17:11:43.881461 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.881434    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwf9n\" (UniqueName: \"kubernetes.io/projected/2a13712f-2bef-4ee2-bc14-21157355d517-kube-api-access-lwf9n\") pod \"feast-operator-controller-manager-6dc7cc57cf-l4pwg\" (UID: \"2a13712f-2bef-4ee2-bc14-21157355d517\") " pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"
May 26 17:11:43.891079 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:43.891054    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"
May 26 17:11:44.019771 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:44.019739    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"]
May 26 17:11:44.022917 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:11:44.022891    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a13712f_2bef_4ee2_bc14_21157355d517.slice/crio-34f00b1a44cf576476a02706e1673c2e06deaa49e4c18abc62a0545c358cefcd WatchSource:0}: Error finding container 34f00b1a44cf576476a02706e1673c2e06deaa49e4c18abc62a0545c358cefcd: Status 404 returned error can't find the container with id 34f00b1a44cf576476a02706e1673c2e06deaa49e4c18abc62a0545c358cefcd
May 26 17:11:44.288294 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:44.288258    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg" event={"ID":"2a13712f-2bef-4ee2-bc14-21157355d517","Type":"ContainerStarted","Data":"34f00b1a44cf576476a02706e1673c2e06deaa49e4c18abc62a0545c358cefcd"}
May 26 17:11:47.300659 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:47.300620    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg" event={"ID":"2a13712f-2bef-4ee2-bc14-21157355d517","Type":"ContainerStarted","Data":"605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4"}
May 26 17:11:47.301105 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:47.300703    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"
May 26 17:11:47.318289 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:47.318243    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg" podStartSLOduration=1.530010791 podStartE2EDuration="4.318229976s" podCreationTimestamp="2026-05-26 17:11:43 +0000 UTC" firstStartedPulling="2026-05-26 17:11:44.024901908 +0000 UTC m=+444.749991387" lastFinishedPulling="2026-05-26 17:11:46.813121077 +0000 UTC m=+447.538210572" observedRunningTime="2026-05-26 17:11:47.315747048 +0000 UTC m=+448.040836548" watchObservedRunningTime="2026-05-26 17:11:47.318229976 +0000 UTC m=+448.043319477"
May 26 17:11:58.306243 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:11:58.306208    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"
May 26 17:12:27.336818 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.336783    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"]
May 26 17:12:27.344868 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.344847    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.347491 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.347453    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 26 17:12:27.347609 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.347526    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-26tfx\""
May 26 17:12:27.347609 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.347572    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 26 17:12:27.347609 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.347573    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 26 17:12:27.347774 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.347756    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 26 17:12:27.347814 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.347788    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 26 17:12:27.348035 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.348001    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 26 17:12:27.348679 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.348661    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 26 17:12:27.351567 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.351544    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"]
May 26 17:12:27.414167 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.414130    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-offline-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.414167 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.414165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-registry-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.414408 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.414187    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-online-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.414408 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.414217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f725405-d3a3-4376-8f4a-ee752d00d69c-feast-data\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.414408 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.414312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.414408 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.414347    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pblrj\" (UniqueName: \"kubernetes.io/projected/1f725405-d3a3-4376-8f4a-ee752d00d69c-kube-api-access-pblrj\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.515366 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.515325    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-offline-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.515366 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.515365    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-registry-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.515387    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-online-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.515408    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f725405-d3a3-4376-8f4a-ee752d00d69c-feast-data\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.515431    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.515451    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pblrj\" (UniqueName: \"kubernetes.io/projected/1f725405-d3a3-4376-8f4a-ee752d00d69c-kube-api-access-pblrj\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:27.515493    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:27.515528    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:27.515493    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:27.515560    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-online-tls podName:1f725405-d3a3-4376-8f4a-ee752d00d69c nodeName:}" failed. No retries permitted until 2026-05-26 17:12:28.015539717 +0000 UTC m=+488.740629195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-online-tls") pod "feast-simple-feast-setup-bb656698-g8sx8" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c") : secret "feast-simple-feast-setup-online-tls" not found
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:27.515599    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-registry-tls podName:1f725405-d3a3-4376-8f4a-ee752d00d69c nodeName:}" failed. No retries permitted until 2026-05-26 17:12:28.015581235 +0000 UTC m=+488.740670714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-registry-tls") pod "feast-simple-feast-setup-bb656698-g8sx8" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c") : secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:27.515615    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls podName:1f725405-d3a3-4376-8f4a-ee752d00d69c nodeName:}" failed. No retries permitted until 2026-05-26 17:12:28.015608071 +0000 UTC m=+488.740697550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls") pod "feast-simple-feast-setup-bb656698-g8sx8" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c") : secret "feast-simple-feast-setup-ui-tls" not found
May 26 17:12:27.515616 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:27.515493    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 26 17:12:27.515997 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:27.515641    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-offline-tls podName:1f725405-d3a3-4376-8f4a-ee752d00d69c nodeName:}" failed. No retries permitted until 2026-05-26 17:12:28.015631876 +0000 UTC m=+488.740721354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-offline-tls") pod "feast-simple-feast-setup-bb656698-g8sx8" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c") : secret "feast-simple-feast-setup-offline-tls" not found
May 26 17:12:27.515997 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.515797    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f725405-d3a3-4376-8f4a-ee752d00d69c-feast-data\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:27.523915 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:27.523893    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pblrj\" (UniqueName: \"kubernetes.io/projected/1f725405-d3a3-4376-8f4a-ee752d00d69c-kube-api-access-pblrj\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:28.020335 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:28.020298    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-offline-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:28.020516 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:28.020352    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-registry-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:28.020516 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:28.020393    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-online-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:28.020516 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:28.020435    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:28.020643 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:28.020570    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 26 17:12:28.020701 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:12:28.020646    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls podName:1f725405-d3a3-4376-8f4a-ee752d00d69c nodeName:}" failed. No retries permitted until 2026-05-26 17:12:29.020627738 +0000 UTC m=+489.745717233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls") pod "feast-simple-feast-setup-bb656698-g8sx8" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c") : secret "feast-simple-feast-setup-ui-tls" not found
May 26 17:12:28.022929 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:28.022901    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-offline-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:28.023380 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:28.023351    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-registry-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:28.023493 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:28.023398    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-online-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:29.030486 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:29.030444    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:29.032901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:29.032879    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls\") pod \"feast-simple-feast-setup-bb656698-g8sx8\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:29.165830 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:29.165795    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:12:29.297411 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:29.297369    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"]
May 26 17:12:29.298860 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:12:29.298832    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f725405_d3a3_4376_8f4a_ee752d00d69c.slice/crio-9ce2e99b920fd11cc08448f57bbd0b0280eb444642b681c7f757094af138b7a0 WatchSource:0}: Error finding container 9ce2e99b920fd11cc08448f57bbd0b0280eb444642b681c7f757094af138b7a0: Status 404 returned error can't find the container with id 9ce2e99b920fd11cc08448f57bbd0b0280eb444642b681c7f757094af138b7a0
May 26 17:12:29.437193 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:12:29.437157    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerStarted","Data":"9ce2e99b920fd11cc08448f57bbd0b0280eb444642b681c7f757094af138b7a0"}
May 26 17:13:02.542067 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:02.542031    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerStarted","Data":"c8ce6e9cb8b528ef3ebe6f64092b5266befdd796ea9e7da0bda231136666ed20"}
May 26 17:13:06.554733 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:06.554692    2571 generic.go:358] "Generic (PLEG): container finished" podID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerID="c8ce6e9cb8b528ef3ebe6f64092b5266befdd796ea9e7da0bda231136666ed20" exitCode=0
May 26 17:13:06.555148 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:06.554764    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerDied","Data":"c8ce6e9cb8b528ef3ebe6f64092b5266befdd796ea9e7da0bda231136666ed20"}
May 26 17:13:07.560096 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:07.560064    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerStarted","Data":"4100be1bd1d4cf75b154186089b4be07dd76e5430d47e44596361f88c1e43252"}
May 26 17:13:12.577389 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:12.577345    2571 generic.go:358] "Generic (PLEG): container finished" podID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerID="4100be1bd1d4cf75b154186089b4be07dd76e5430d47e44596361f88c1e43252" exitCode=0
May 26 17:13:12.577860 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:12.577395    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerDied","Data":"4100be1bd1d4cf75b154186089b4be07dd76e5430d47e44596361f88c1e43252"}
May 26 17:13:13.585732 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:13.585688    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerStarted","Data":"f693d5c15077a349729a25d1b8dbc6815aa60faf6815423eab6eab0a7149be61"}
May 26 17:13:13.586246 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:13.585740    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerStarted","Data":"486467f0fe3d1008966d3924b011c41398e428606453bfd14a5d0e85a1f0497a"}
May 26 17:13:13.586246 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:13.585758    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerStarted","Data":"532afea26c04e7a87e4d39ceb05ac689e9f6c4cfcef52efd1408bcd208cf5181"}
May 26 17:13:13.586246 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:13.585772    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerStarted","Data":"3de2275ba16199b0814111fb14dc4e792b261509d67f5c820941cf53ff229b38"}
May 26 17:13:13.610741 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:13.610676    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podStartSLOduration=14.346623731 podStartE2EDuration="46.610656672s" podCreationTimestamp="2026-05-26 17:12:27 +0000 UTC" firstStartedPulling="2026-05-26 17:12:29.300548735 +0000 UTC m=+490.025638215" lastFinishedPulling="2026-05-26 17:13:01.564581662 +0000 UTC m=+522.289671156" observedRunningTime="2026-05-26 17:13:13.609867331 +0000 UTC m=+534.334956832" watchObservedRunningTime="2026-05-26 17:13:13.610656672 +0000 UTC m=+534.335746173"
May 26 17:13:14.166983 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:14.166945    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:14.167201 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:14.166995    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:14.167201 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:14.167028    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:14.167201 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:14.167042    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:14.169196 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:14.169150    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused"
May 26 17:13:14.169524 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:14.169198    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused"
May 26 17:13:14.169524 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:14.169194    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 26 17:13:14.169524 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:14.169213    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused"
May 26 17:13:17.166422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:17.166357    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused"
May 26 17:13:17.166904 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:17.166357    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused"
May 26 17:13:17.166904 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:17.166371    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 26 17:13:17.166904 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:17.166357    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused"
May 26 17:13:19.166628 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:19.166592    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:19.166628 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:19.166639    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:19.167091 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:19.166649    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:19.167091 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:19.166656    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:20.167471 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:20.167441    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:20.167960 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:20.167713    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:20.168151 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:20.168126    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:20.168296 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:20.168267    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:20.168363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:20.168311    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:20.172162 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:20.172142    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:20.611480 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:20.611446    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:13:20.613652 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:13:20.613627    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:14:19.759641 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:19.759611    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:14:19.759641 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:19.759632    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:14:22.691520 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:22.691481    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"]
May 26 17:14:22.691979 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:22.691946    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="registry" containerID="cri-o://3de2275ba16199b0814111fb14dc4e792b261509d67f5c820941cf53ff229b38" gracePeriod=30
May 26 17:14:22.692401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:22.692102    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="offline" containerID="cri-o://486467f0fe3d1008966d3924b011c41398e428606453bfd14a5d0e85a1f0497a" gracePeriod=30
May 26 17:14:22.692401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:22.692173    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="ui" containerID="cri-o://f693d5c15077a349729a25d1b8dbc6815aa60faf6815423eab6eab0a7149be61" gracePeriod=30
May 26 17:14:22.692401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:22.692281    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="online" containerID="cri-o://532afea26c04e7a87e4d39ceb05ac689e9f6c4cfcef52efd1408bcd208cf5181" gracePeriod=30
May 26 17:14:22.974797 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:22.974706    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"]
May 26 17:14:22.978286 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:22.978263    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:22.981784 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:22.981761    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-m86kn\""
May 26 17:14:22.989799 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:22.989765    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"]
May 26 17:14:23.117444 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.117395    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4nd6\" (UniqueName: \"kubernetes.io/projected/ed31bff7-927f-4978-967a-1dcf00328330-kube-api-access-w4nd6\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.117658 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.117458    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-offline-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.117658 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.117493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-online-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.117658 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.117596    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ed31bff7-927f-4978-967a-1dcf00328330-feast-data\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.117658 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.117653    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-registry-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.117813 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.117725    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-ui-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.218908 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.218863    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ed31bff7-927f-4978-967a-1dcf00328330-feast-data\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.219126 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.218924    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-registry-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.219126 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.218958    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-ui-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.219126 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.219039    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w4nd6\" (UniqueName: \"kubernetes.io/projected/ed31bff7-927f-4978-967a-1dcf00328330-kube-api-access-w4nd6\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.219126 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.219071    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-offline-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.219126 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:14:23.219088    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:14:23.219126 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.219099    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-online-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.219433 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:14:23.219169    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-registry-tls podName:ed31bff7-927f-4978-967a-1dcf00328330 nodeName:}" failed. No retries permitted until 2026-05-26 17:14:23.719142901 +0000 UTC m=+604.444232402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-registry-tls") pod "feast-simple-feast-setup-bb656698-djxbk" (UID: "ed31bff7-927f-4978-967a-1dcf00328330") : secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:14:23.219433 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:14:23.219206    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 26 17:14:23.219433 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:14:23.219253    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-online-tls podName:ed31bff7-927f-4978-967a-1dcf00328330 nodeName:}" failed. No retries permitted until 2026-05-26 17:14:23.719237761 +0000 UTC m=+604.444327243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-online-tls") pod "feast-simple-feast-setup-bb656698-djxbk" (UID: "ed31bff7-927f-4978-967a-1dcf00328330") : secret "feast-simple-feast-setup-online-tls" not found
May 26 17:14:23.219433 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:14:23.219305    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 26 17:14:23.219433 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:14:23.219343    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-offline-tls podName:ed31bff7-927f-4978-967a-1dcf00328330 nodeName:}" failed. No retries permitted until 2026-05-26 17:14:23.719332896 +0000 UTC m=+604.444422386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-offline-tls") pod "feast-simple-feast-setup-bb656698-djxbk" (UID: "ed31bff7-927f-4978-967a-1dcf00328330") : secret "feast-simple-feast-setup-offline-tls" not found
May 26 17:14:23.219433 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.219342    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ed31bff7-927f-4978-967a-1dcf00328330-feast-data\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.221760 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.221727    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-ui-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.232424 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.232353    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4nd6\" (UniqueName: \"kubernetes.io/projected/ed31bff7-927f-4978-967a-1dcf00328330-kube-api-access-w4nd6\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.723328 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.723283    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-offline-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.723328 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.723327    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-online-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.723773 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.723370    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-registry-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.725867 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.725840    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-offline-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.725995 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.725939    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-registry-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.725995 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.725975    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-online-tls\") pod \"feast-simple-feast-setup-bb656698-djxbk\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") " pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:23.820165 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.820080    2571 generic.go:358] "Generic (PLEG): container finished" podID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerID="f693d5c15077a349729a25d1b8dbc6815aa60faf6815423eab6eab0a7149be61" exitCode=0
May 26 17:14:23.820165 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.820109    2571 generic.go:358] "Generic (PLEG): container finished" podID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerID="532afea26c04e7a87e4d39ceb05ac689e9f6c4cfcef52efd1408bcd208cf5181" exitCode=0
May 26 17:14:23.820165 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.820140    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerDied","Data":"f693d5c15077a349729a25d1b8dbc6815aa60faf6815423eab6eab0a7149be61"}
May 26 17:14:23.820368 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.820175    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerDied","Data":"532afea26c04e7a87e4d39ceb05ac689e9f6c4cfcef52efd1408bcd208cf5181"}
May 26 17:14:23.888189 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:23.888155    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:24.022813 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:24.022785    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"]
May 26 17:14:24.024392 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:14:24.024365    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded31bff7_927f_4978_967a_1dcf00328330.slice/crio-ebd247aba5c16cba67c45205a6a503fbb0248f4ab8bd2f4cefee71ecd416f427 WatchSource:0}: Error finding container ebd247aba5c16cba67c45205a6a503fbb0248f4ab8bd2f4cefee71ecd416f427: Status 404 returned error can't find the container with id ebd247aba5c16cba67c45205a6a503fbb0248f4ab8bd2f4cefee71ecd416f427
May 26 17:14:24.828715 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:24.828673    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerStarted","Data":"b8abe972bff55bf53fe181659cf83454a383c29329388fc1b529dff16e3a3103"}
May 26 17:14:24.828715 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:24.828712    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerStarted","Data":"ebd247aba5c16cba67c45205a6a503fbb0248f4ab8bd2f4cefee71ecd416f427"}
May 26 17:14:28.845748 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:28.845713    2571 generic.go:358] "Generic (PLEG): container finished" podID="ed31bff7-927f-4978-967a-1dcf00328330" containerID="b8abe972bff55bf53fe181659cf83454a383c29329388fc1b529dff16e3a3103" exitCode=0
May 26 17:14:28.846104 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:28.845774    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerDied","Data":"b8abe972bff55bf53fe181659cf83454a383c29329388fc1b529dff16e3a3103"}
May 26 17:14:29.850792 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:29.850757    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerStarted","Data":"3d6868df34461ff1b5372604f0068ba072f6c949150a576d2227b339c927a3ca"}
May 26 17:14:30.611357 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:30.611308    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused"
May 26 17:14:30.611357 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:30.611337    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 26 17:14:34.869998 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:34.869912    2571 generic.go:358] "Generic (PLEG): container finished" podID="ed31bff7-927f-4978-967a-1dcf00328330" containerID="3d6868df34461ff1b5372604f0068ba072f6c949150a576d2227b339c927a3ca" exitCode=0
May 26 17:14:34.869998 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:34.869980    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerDied","Data":"3d6868df34461ff1b5372604f0068ba072f6c949150a576d2227b339c927a3ca"}
May 26 17:14:35.878284 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.878241    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerStarted","Data":"23f100cc60bcf47fa869cbad5526549202ac55b5c95842e29cc53216a0532d43"}
May 26 17:14:35.878770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.878298    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerStarted","Data":"36b1f8c5d37739ea4809c46168559d78097e128be90d6f01c416fd792403090f"}
May 26 17:14:35.878770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.878315    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerStarted","Data":"4237debbb02f710d7dca2d21acf369956ecb12507ea3a541b72170b0f3a81201"}
May 26 17:14:35.878770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.878329    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerStarted","Data":"04031d90a4a2b4178f333656f63ce62f2e584838a073845a49af6415dc0e7926"}
May 26 17:14:35.889311 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.889262    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:35.889311 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.889314    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:35.889559 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.889331    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:35.889559 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.889351    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:35.891818 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.891775    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused"
May 26 17:14:35.891978 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.891819    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused"
May 26 17:14:35.891978 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.891908    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused"
May 26 17:14:35.892204 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.892171    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 26 17:14:35.905598 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:35.905537    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podStartSLOduration=13.905517455 podStartE2EDuration="13.905517455s" podCreationTimestamp="2026-05-26 17:14:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:14:35.904094313 +0000 UTC m=+616.629183814" watchObservedRunningTime="2026-05-26 17:14:35.905517455 +0000 UTC m=+616.630606959"
May 26 17:14:38.888810 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:38.888765    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused"
May 26 17:14:38.891449 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:38.889084    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 26 17:14:38.891449 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:38.888758    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused"
May 26 17:14:38.891449 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:38.889529    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused"
May 26 17:14:40.611479 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:40.611424    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused"
May 26 17:14:40.611971 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:40.611498    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 26 17:14:41.889246 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.889213    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.889701 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.889382    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.889771 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.889714    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.889856 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.889834    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.889924 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.889869    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.889924 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.889885    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.890235 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.890215    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.890329 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.890259    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.890329 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.890286    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.893816 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.893796    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.900993 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.900971    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:41.903746 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:41.903724    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:14:50.611570 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:50.611525    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused"
May 26 17:14:50.611964 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:50.611530    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 26 17:14:50.611964 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:50.611656    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:14:50.611964 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:50.611769    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:14:52.941671 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:52.941638    2571 generic.go:358] "Generic (PLEG): container finished" podID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerID="486467f0fe3d1008966d3924b011c41398e428606453bfd14a5d0e85a1f0497a" exitCode=137
May 26 17:14:52.941671 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:52.941662    2571 generic.go:358] "Generic (PLEG): container finished" podID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerID="3de2275ba16199b0814111fb14dc4e792b261509d67f5c820941cf53ff229b38" exitCode=137
May 26 17:14:52.942124 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:52.941708    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerDied","Data":"486467f0fe3d1008966d3924b011c41398e428606453bfd14a5d0e85a1f0497a"}
May 26 17:14:52.942124 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:52.941740    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerDied","Data":"3de2275ba16199b0814111fb14dc4e792b261509d67f5c820941cf53ff229b38"}
May 26 17:14:53.357870 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.357846    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:14:53.394619 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.394590    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-offline-tls\") pod \"1f725405-d3a3-4376-8f4a-ee752d00d69c\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") "
May 26 17:14:53.394799 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.394627    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pblrj\" (UniqueName: \"kubernetes.io/projected/1f725405-d3a3-4376-8f4a-ee752d00d69c-kube-api-access-pblrj\") pod \"1f725405-d3a3-4376-8f4a-ee752d00d69c\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") "
May 26 17:14:53.394799 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.394668    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls\") pod \"1f725405-d3a3-4376-8f4a-ee752d00d69c\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") "
May 26 17:14:53.394799 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.394685    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-registry-tls\") pod \"1f725405-d3a3-4376-8f4a-ee752d00d69c\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") "
May 26 17:14:53.394799 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.394727    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f725405-d3a3-4376-8f4a-ee752d00d69c-feast-data\") pod \"1f725405-d3a3-4376-8f4a-ee752d00d69c\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") "
May 26 17:14:53.395041 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.394901    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-online-tls\") pod \"1f725405-d3a3-4376-8f4a-ee752d00d69c\" (UID: \"1f725405-d3a3-4376-8f4a-ee752d00d69c\") "
May 26 17:14:53.395463 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.395270    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f725405-d3a3-4376-8f4a-ee752d00d69c-feast-data" (OuterVolumeSpecName: "feast-data") pod "1f725405-d3a3-4376-8f4a-ee752d00d69c" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:14:53.395463 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.395412    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f725405-d3a3-4376-8f4a-ee752d00d69c-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:14:53.397362 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.397187    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f725405-d3a3-4376-8f4a-ee752d00d69c-kube-api-access-pblrj" (OuterVolumeSpecName: "kube-api-access-pblrj") pod "1f725405-d3a3-4376-8f4a-ee752d00d69c" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c"). InnerVolumeSpecName "kube-api-access-pblrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:14:53.397362 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.397238    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1f725405-d3a3-4376-8f4a-ee752d00d69c" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:14:53.397362 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.397280    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1f725405-d3a3-4376-8f4a-ee752d00d69c" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:14:53.397586 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.397462    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-online-tls" (OuterVolumeSpecName: "online-tls") pod "1f725405-d3a3-4376-8f4a-ee752d00d69c" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:14:53.397586 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.397518    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1f725405-d3a3-4376-8f4a-ee752d00d69c" (UID: "1f725405-d3a3-4376-8f4a-ee752d00d69c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:14:53.496740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.496700    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:14:53.496740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.496729    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:14:53.496740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.496739    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pblrj\" (UniqueName: \"kubernetes.io/projected/1f725405-d3a3-4376-8f4a-ee752d00d69c-kube-api-access-pblrj\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:14:53.496740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.496748    2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-ui-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:14:53.497062 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.496757    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f725405-d3a3-4376-8f4a-ee752d00d69c-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:14:53.947370 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.947345    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"
May 26 17:14:53.947764 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.947342    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8" event={"ID":"1f725405-d3a3-4376-8f4a-ee752d00d69c","Type":"ContainerDied","Data":"9ce2e99b920fd11cc08448f57bbd0b0280eb444642b681c7f757094af138b7a0"}
May 26 17:14:53.947764 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.947477    2571 scope.go:117] "RemoveContainer" containerID="f693d5c15077a349729a25d1b8dbc6815aa60faf6815423eab6eab0a7149be61"
May 26 17:14:53.956392 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.956370    2571 scope.go:117] "RemoveContainer" containerID="486467f0fe3d1008966d3924b011c41398e428606453bfd14a5d0e85a1f0497a"
May 26 17:14:53.964361 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.964342    2571 scope.go:117] "RemoveContainer" containerID="532afea26c04e7a87e4d39ceb05ac689e9f6c4cfcef52efd1408bcd208cf5181"
May 26 17:14:53.969672 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.969238    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"]
May 26 17:14:53.977059 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.973032    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-g8sx8"]
May 26 17:14:53.977059 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.974588    2571 scope.go:117] "RemoveContainer" containerID="3de2275ba16199b0814111fb14dc4e792b261509d67f5c820941cf53ff229b38"
May 26 17:14:53.984614 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.984584    2571 scope.go:117] "RemoveContainer" containerID="4100be1bd1d4cf75b154186089b4be07dd76e5430d47e44596361f88c1e43252"
May 26 17:14:53.992193 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:53.992174    2571 scope.go:117] "RemoveContainer" containerID="c8ce6e9cb8b528ef3ebe6f64092b5266befdd796ea9e7da0bda231136666ed20"
May 26 17:14:55.834722 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:14:55.834685    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" path="/var/lib/kubelet/pods/1f725405-d3a3-4376-8f4a-ee752d00d69c/volumes"
May 26 17:15:44.678738 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.678701    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"]
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679088    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="feast-init"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679100    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="feast-init"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679109    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="feast-apply"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679114    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="feast-apply"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679121    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="registry"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679127    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="registry"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679140    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="offline"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679145    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="offline"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679155    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="online"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679160    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="online"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679167    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="ui"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679171    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="ui"
May 26 17:15:44.679219 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679226    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="ui"
May 26 17:15:44.679602 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679235    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="offline"
May 26 17:15:44.679602 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679243    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="registry"
May 26 17:15:44.679602 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.679252    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="1f725405-d3a3-4376-8f4a-ee752d00d69c" containerName="online"
May 26 17:15:44.682240 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.682225    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.684968 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.684941    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 26 17:15:44.684968 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.684960    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 26 17:15:44.686152 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.686126    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 26 17:15:44.686287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.686180    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-dqqxq\""
May 26 17:15:44.686287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.686128    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 26 17:15:44.686287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.686206    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 26 17:15:44.686287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.686128    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 26 17:15:44.686287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.686266    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 26 17:15:44.692460 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.692433    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"]
May 26 17:15:44.742932 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.742892    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-offline-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.743168 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.742950    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-online-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.743168 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.742986    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6aaef280-cff6-4b47-8999-4f90926ec19c-registry-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.743168 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.743043    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaef280-cff6-4b47-8999-4f90926ec19c-feast-data\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.743168 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.743117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-ui-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.743168 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.743136    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlgtd\" (UniqueName: \"kubernetes.io/projected/6aaef280-cff6-4b47-8999-4f90926ec19c-kube-api-access-qlgtd\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.843688 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.843651    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-online-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.843688 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.843694    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6aaef280-cff6-4b47-8999-4f90926ec19c-registry-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.843948 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.843818    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaef280-cff6-4b47-8999-4f90926ec19c-feast-data\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.843948 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.843880    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-ui-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.843948 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.843904    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgtd\" (UniqueName: \"kubernetes.io/projected/6aaef280-cff6-4b47-8999-4f90926ec19c-kube-api-access-qlgtd\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.844247 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.844222    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-offline-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.844339 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.844225    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaef280-cff6-4b47-8999-4f90926ec19c-feast-data\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.844761 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.844733    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6aaef280-cff6-4b47-8999-4f90926ec19c-registry-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.846397 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.846376    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-online-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.846485 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.846411    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-ui-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.846485 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.846477    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-offline-tls\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.852339 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.852322    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgtd\" (UniqueName: \"kubernetes.io/projected/6aaef280-cff6-4b47-8999-4f90926ec19c-kube-api-access-qlgtd\") pod \"feast-simple-feast-remote-setup-744f75848f-6jg4p\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:44.993141 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:44.993102    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:45.128145 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:45.128121    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"]
May 26 17:15:45.130509 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:15:45.130479    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6aaef280_cff6_4b47_8999_4f90926ec19c.slice/crio-e9b03f8b5e7b90425c9b51f9ee184215a49d83009f15018a9d77b35dfb4d6285 WatchSource:0}: Error finding container e9b03f8b5e7b90425c9b51f9ee184215a49d83009f15018a9d77b35dfb4d6285: Status 404 returned error can't find the container with id e9b03f8b5e7b90425c9b51f9ee184215a49d83009f15018a9d77b35dfb4d6285
May 26 17:15:45.132371 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:45.132350    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 17:15:46.124512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:46.124477    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerStarted","Data":"d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59"}
May 26 17:15:46.124512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:46.124511    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerStarted","Data":"e9b03f8b5e7b90425c9b51f9ee184215a49d83009f15018a9d77b35dfb4d6285"}
May 26 17:15:50.139713 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:50.139625    2571 generic.go:358] "Generic (PLEG): container finished" podID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerID="d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59" exitCode=0
May 26 17:15:50.139713 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:50.139681    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerDied","Data":"d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59"}
May 26 17:15:51.144472 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:51.144440    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerStarted","Data":"272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e"}
May 26 17:15:56.163939 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:56.163842    2571 generic.go:358] "Generic (PLEG): container finished" podID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerID="272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e" exitCode=0
May 26 17:15:56.163939 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:56.163918    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerDied","Data":"272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e"}
May 26 17:15:57.171561 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:57.171517    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerStarted","Data":"6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528"}
May 26 17:15:57.171561 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:57.171562    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerStarted","Data":"45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b"}
May 26 17:15:57.172164 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:57.171573    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerStarted","Data":"77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563"}
May 26 17:15:57.198591 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:57.198527    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podStartSLOduration=13.198508228 podStartE2EDuration="13.198508228s" podCreationTimestamp="2026-05-26 17:15:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:15:57.195461466 +0000 UTC m=+697.920551001" watchObservedRunningTime="2026-05-26 17:15:57.198508228 +0000 UTC m=+697.923597730"
May 26 17:15:59.993585 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:59.993542    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:59.994062 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:59.993596    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:59.994062 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:59.993611    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:15:59.994738 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:59.994703    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused"
May 26 17:15:59.994839 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:59.994737    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused"
May 26 17:15:59.994839 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:15:59.994753    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 26 17:16:02.994511 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:16:02.994477    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:16:02.994971 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:16:02.994749    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:16:02.994971 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:16:02.994850    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:16:02.994971 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:16:02.994876    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:16:02.995173 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:16:02.995145    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:16:02.995235 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:16:02.995202    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:16:02.998800 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:16:02.998785    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:16:03.193839 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:16:03.193805    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:16:03.196773 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:16:03.196750    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:17:05.905856 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:05.905819    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"]
May 26 17:17:05.906396 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:05.906272    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="online" containerID="cri-o://77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563" gracePeriod=30
May 26 17:17:05.906396 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:05.906329    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="offline" containerID="cri-o://45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b" gracePeriod=30
May 26 17:17:05.906519 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:05.906339    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="ui" containerID="cri-o://6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528" gracePeriod=30
May 26 17:17:06.021175 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:06.021144    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"]
May 26 17:17:06.021720 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:06.021508    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="registry" containerID="cri-o://04031d90a4a2b4178f333656f63ce62f2e584838a073845a49af6415dc0e7926" gracePeriod=30
May 26 17:17:06.021720 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:06.021565    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="ui" containerID="cri-o://23f100cc60bcf47fa869cbad5526549202ac55b5c95842e29cc53216a0532d43" gracePeriod=30
May 26 17:17:06.021720 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:06.021566    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="offline" containerID="cri-o://36b1f8c5d37739ea4809c46168559d78097e128be90d6f01c416fd792403090f" gracePeriod=30
May 26 17:17:06.021720 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:06.021604    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="online" containerID="cri-o://4237debbb02f710d7dca2d21acf369956ecb12507ea3a541b72170b0f3a81201" gracePeriod=30
May 26 17:17:07.420667 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:07.420632    2571 generic.go:358] "Generic (PLEG): container finished" podID="ed31bff7-927f-4978-967a-1dcf00328330" containerID="23f100cc60bcf47fa869cbad5526549202ac55b5c95842e29cc53216a0532d43" exitCode=0
May 26 17:17:07.420667 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:07.420658    2571 generic.go:358] "Generic (PLEG): container finished" podID="ed31bff7-927f-4978-967a-1dcf00328330" containerID="4237debbb02f710d7dca2d21acf369956ecb12507ea3a541b72170b0f3a81201" exitCode=0
May 26 17:17:07.421160 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:07.420704    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerDied","Data":"23f100cc60bcf47fa869cbad5526549202ac55b5c95842e29cc53216a0532d43"}
May 26 17:17:07.421160 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:07.420736    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerDied","Data":"4237debbb02f710d7dca2d21acf369956ecb12507ea3a541b72170b0f3a81201"}
May 26 17:17:07.422810 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:07.422788    2571 generic.go:358] "Generic (PLEG): container finished" podID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerID="6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528" exitCode=0
May 26 17:17:07.422810 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:07.422806    2571 generic.go:358] "Generic (PLEG): container finished" podID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerID="77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563" exitCode=0
May 26 17:17:07.422953 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:07.422824    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerDied","Data":"6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528"}
May 26 17:17:07.422953 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:07.422841    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerDied","Data":"77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563"}
May 26 17:17:11.890696 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:11.890635    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused"
May 26 17:17:11.901141 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:11.901108    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 26 17:17:12.995359 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:12.995310    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused"
May 26 17:17:13.194435 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:13.194393    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 26 17:17:21.890057 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:21.889984    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused"
May 26 17:17:21.901783 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:21.901746    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 26 17:17:22.995978 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:22.995932    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused"
May 26 17:17:23.194698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:23.194654    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 26 17:17:31.890735 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:31.890690    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused"
May 26 17:17:31.891211 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:31.890823    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:17:31.901166 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:31.901137    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 26 17:17:31.901241 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:31.901234    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:17:32.995233 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:32.995193    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused"
May 26 17:17:32.995734 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:32.995380    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:17:33.194287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:33.194247    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 26 17:17:33.194462 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:33.194384    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:17:36.139173 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.139141    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:17:36.225425 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.225316    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlgtd\" (UniqueName: \"kubernetes.io/projected/6aaef280-cff6-4b47-8999-4f90926ec19c-kube-api-access-qlgtd\") pod \"6aaef280-cff6-4b47-8999-4f90926ec19c\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") "
May 26 17:17:36.225425 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.225398    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-online-tls\") pod \"6aaef280-cff6-4b47-8999-4f90926ec19c\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") "
May 26 17:17:36.225671 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.225570    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6aaef280-cff6-4b47-8999-4f90926ec19c-registry-tls\") pod \"6aaef280-cff6-4b47-8999-4f90926ec19c\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") "
May 26 17:17:36.225671 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.225628    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaef280-cff6-4b47-8999-4f90926ec19c-feast-data\") pod \"6aaef280-cff6-4b47-8999-4f90926ec19c\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") "
May 26 17:17:36.225671 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.225660    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-offline-tls\") pod \"6aaef280-cff6-4b47-8999-4f90926ec19c\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") "
May 26 17:17:36.225813 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.225702    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-ui-tls\") pod \"6aaef280-cff6-4b47-8999-4f90926ec19c\" (UID: \"6aaef280-cff6-4b47-8999-4f90926ec19c\") "
May 26 17:17:36.225968 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.225934    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aaef280-cff6-4b47-8999-4f90926ec19c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6aaef280-cff6-4b47-8999-4f90926ec19c" (UID: "6aaef280-cff6-4b47-8999-4f90926ec19c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:17:36.226171 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.226147    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaef280-cff6-4b47-8999-4f90926ec19c-feast-data" (OuterVolumeSpecName: "feast-data") pod "6aaef280-cff6-4b47-8999-4f90926ec19c" (UID: "6aaef280-cff6-4b47-8999-4f90926ec19c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:17:36.227629 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.227606    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6aaef280-cff6-4b47-8999-4f90926ec19c" (UID: "6aaef280-cff6-4b47-8999-4f90926ec19c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:17:36.227734 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.227680    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aaef280-cff6-4b47-8999-4f90926ec19c-kube-api-access-qlgtd" (OuterVolumeSpecName: "kube-api-access-qlgtd") pod "6aaef280-cff6-4b47-8999-4f90926ec19c" (UID: "6aaef280-cff6-4b47-8999-4f90926ec19c"). InnerVolumeSpecName "kube-api-access-qlgtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:17:36.227734 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.227715    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-online-tls" (OuterVolumeSpecName: "online-tls") pod "6aaef280-cff6-4b47-8999-4f90926ec19c" (UID: "6aaef280-cff6-4b47-8999-4f90926ec19c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:17:36.227806 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.227735    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "6aaef280-cff6-4b47-8999-4f90926ec19c" (UID: "6aaef280-cff6-4b47-8999-4f90926ec19c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:17:36.326990 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.326953    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlgtd\" (UniqueName: \"kubernetes.io/projected/6aaef280-cff6-4b47-8999-4f90926ec19c-kube-api-access-qlgtd\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.326990 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.326984    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.326990 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.326994    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6aaef280-cff6-4b47-8999-4f90926ec19c-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.327240 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.327002    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaef280-cff6-4b47-8999-4f90926ec19c-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.327240 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.327030    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.327240 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.327040    2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6aaef280-cff6-4b47-8999-4f90926ec19c-ui-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.522259 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.522224    2571 generic.go:358] "Generic (PLEG): container finished" podID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerID="45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b" exitCode=137
May 26 17:17:36.522433 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.522299    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerDied","Data":"45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b"}
May 26 17:17:36.522433 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.522310    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"
May 26 17:17:36.522433 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.522340    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p" event={"ID":"6aaef280-cff6-4b47-8999-4f90926ec19c","Type":"ContainerDied","Data":"e9b03f8b5e7b90425c9b51f9ee184215a49d83009f15018a9d77b35dfb4d6285"}
May 26 17:17:36.522433 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.522357    2571 scope.go:117] "RemoveContainer" containerID="6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528"
May 26 17:17:36.525350 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.525320    2571 generic.go:358] "Generic (PLEG): container finished" podID="ed31bff7-927f-4978-967a-1dcf00328330" containerID="36b1f8c5d37739ea4809c46168559d78097e128be90d6f01c416fd792403090f" exitCode=137
May 26 17:17:36.525350 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.525344    2571 generic.go:358] "Generic (PLEG): container finished" podID="ed31bff7-927f-4978-967a-1dcf00328330" containerID="04031d90a4a2b4178f333656f63ce62f2e584838a073845a49af6415dc0e7926" exitCode=137
May 26 17:17:36.525530 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.525377    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerDied","Data":"36b1f8c5d37739ea4809c46168559d78097e128be90d6f01c416fd792403090f"}
May 26 17:17:36.525530 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.525405    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerDied","Data":"04031d90a4a2b4178f333656f63ce62f2e584838a073845a49af6415dc0e7926"}
May 26 17:17:36.531695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.531678    2571 scope.go:117] "RemoveContainer" containerID="45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b"
May 26 17:17:36.539891 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.539872    2571 scope.go:117] "RemoveContainer" containerID="77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563"
May 26 17:17:36.545469 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.545443    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"]
May 26 17:17:36.548749 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.548725    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-744f75848f-6jg4p"]
May 26 17:17:36.549760 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.549746    2571 scope.go:117] "RemoveContainer" containerID="272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e"
May 26 17:17:36.557844 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.557826    2571 scope.go:117] "RemoveContainer" containerID="d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59"
May 26 17:17:36.565187 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.565163    2571 scope.go:117] "RemoveContainer" containerID="6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528"
May 26 17:17:36.565468 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:17:36.565443    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528\": container with ID starting with 6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528 not found: ID does not exist" containerID="6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528"
May 26 17:17:36.565535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.565482    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528"} err="failed to get container status \"6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528\": rpc error: code = NotFound desc = could not find container \"6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528\": container with ID starting with 6d9bde76d78f2ed768d4883cefe710e8f5676c74344da3e7deca1b4038603528 not found: ID does not exist"
May 26 17:17:36.565535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.565509    2571 scope.go:117] "RemoveContainer" containerID="45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b"
May 26 17:17:36.565760 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:17:36.565741    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b\": container with ID starting with 45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b not found: ID does not exist" containerID="45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b"
May 26 17:17:36.565811 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.565769    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b"} err="failed to get container status \"45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b\": rpc error: code = NotFound desc = could not find container \"45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b\": container with ID starting with 45a60d859ba37c6efafda562431157acf5b6b1a17e3ce84ef65f4a15cd70742b not found: ID does not exist"
May 26 17:17:36.565811 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.565791    2571 scope.go:117] "RemoveContainer" containerID="77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563"
May 26 17:17:36.566070 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:17:36.566040    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563\": container with ID starting with 77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563 not found: ID does not exist" containerID="77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563"
May 26 17:17:36.566154 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.566067    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563"} err="failed to get container status \"77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563\": rpc error: code = NotFound desc = could not find container \"77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563\": container with ID starting with 77900a02ef595dceafcabfc0528f95bf1f668830edb044881e375fac20c78563 not found: ID does not exist"
May 26 17:17:36.566154 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.566088    2571 scope.go:117] "RemoveContainer" containerID="272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e"
May 26 17:17:36.566338 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:17:36.566320    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e\": container with ID starting with 272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e not found: ID does not exist" containerID="272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e"
May 26 17:17:36.566375 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.566343    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e"} err="failed to get container status \"272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e\": rpc error: code = NotFound desc = could not find container \"272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e\": container with ID starting with 272d278cd6dfd3569f00cfd70d90b0bb9acdd4afa728c8d3e2529afd1ae12e4e not found: ID does not exist"
May 26 17:17:36.566375 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.566359    2571 scope.go:117] "RemoveContainer" containerID="d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59"
May 26 17:17:36.566568 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:17:36.566552    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59\": container with ID starting with d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59 not found: ID does not exist" containerID="d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59"
May 26 17:17:36.566609 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.566572    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59"} err="failed to get container status \"d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59\": rpc error: code = NotFound desc = could not find container \"d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59\": container with ID starting with d4311aa991b373e649a7a150d57d88a84ee40efa6b59d6320af922a755db5c59 not found: ID does not exist"
May 26 17:17:36.669529 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.669506    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:17:36.730415 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.730380    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-registry-tls\") pod \"ed31bff7-927f-4978-967a-1dcf00328330\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") "
May 26 17:17:36.730581 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.730427    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ed31bff7-927f-4978-967a-1dcf00328330-feast-data\") pod \"ed31bff7-927f-4978-967a-1dcf00328330\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") "
May 26 17:17:36.730581 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.730533    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-offline-tls\") pod \"ed31bff7-927f-4978-967a-1dcf00328330\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") "
May 26 17:17:36.730581 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.730572    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-ui-tls\") pod \"ed31bff7-927f-4978-967a-1dcf00328330\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") "
May 26 17:17:36.730749 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.730650    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4nd6\" (UniqueName: \"kubernetes.io/projected/ed31bff7-927f-4978-967a-1dcf00328330-kube-api-access-w4nd6\") pod \"ed31bff7-927f-4978-967a-1dcf00328330\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") "
May 26 17:17:36.730749 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.730677    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-online-tls\") pod \"ed31bff7-927f-4978-967a-1dcf00328330\" (UID: \"ed31bff7-927f-4978-967a-1dcf00328330\") "
May 26 17:17:36.730921 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.730897    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed31bff7-927f-4978-967a-1dcf00328330-feast-data" (OuterVolumeSpecName: "feast-data") pod "ed31bff7-927f-4978-967a-1dcf00328330" (UID: "ed31bff7-927f-4978-967a-1dcf00328330"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:17:36.732703 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.732672    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "ed31bff7-927f-4978-967a-1dcf00328330" (UID: "ed31bff7-927f-4978-967a-1dcf00328330"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:17:36.732703 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.732692    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ed31bff7-927f-4978-967a-1dcf00328330" (UID: "ed31bff7-927f-4978-967a-1dcf00328330"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:17:36.732857 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.732769    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-online-tls" (OuterVolumeSpecName: "online-tls") pod "ed31bff7-927f-4978-967a-1dcf00328330" (UID: "ed31bff7-927f-4978-967a-1dcf00328330"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:17:36.733166 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.733145    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed31bff7-927f-4978-967a-1dcf00328330-kube-api-access-w4nd6" (OuterVolumeSpecName: "kube-api-access-w4nd6") pod "ed31bff7-927f-4978-967a-1dcf00328330" (UID: "ed31bff7-927f-4978-967a-1dcf00328330"). InnerVolumeSpecName "kube-api-access-w4nd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:17:36.733166 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.733151    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ed31bff7-927f-4978-967a-1dcf00328330" (UID: "ed31bff7-927f-4978-967a-1dcf00328330"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:17:36.831422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.831339    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4nd6\" (UniqueName: \"kubernetes.io/projected/ed31bff7-927f-4978-967a-1dcf00328330-kube-api-access-w4nd6\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.831422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.831371    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.831422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.831382    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.831422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.831390    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ed31bff7-927f-4978-967a-1dcf00328330-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.831422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.831399    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:36.831422 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:36.831407    2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ed31bff7-927f-4978-967a-1dcf00328330-ui-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:17:37.530877 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.530833    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk" event={"ID":"ed31bff7-927f-4978-967a-1dcf00328330","Type":"ContainerDied","Data":"ebd247aba5c16cba67c45205a6a503fbb0248f4ab8bd2f4cefee71ecd416f427"}
May 26 17:17:37.531381 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.530893    2571 scope.go:117] "RemoveContainer" containerID="23f100cc60bcf47fa869cbad5526549202ac55b5c95842e29cc53216a0532d43"
May 26 17:17:37.531381 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.530843    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"
May 26 17:17:37.539464 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.539441    2571 scope.go:117] "RemoveContainer" containerID="36b1f8c5d37739ea4809c46168559d78097e128be90d6f01c416fd792403090f"
May 26 17:17:37.547119 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.547099    2571 scope.go:117] "RemoveContainer" containerID="4237debbb02f710d7dca2d21acf369956ecb12507ea3a541b72170b0f3a81201"
May 26 17:17:37.553492 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.553469    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"]
May 26 17:17:37.555217 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.555193    2571 scope.go:117] "RemoveContainer" containerID="04031d90a4a2b4178f333656f63ce62f2e584838a073845a49af6415dc0e7926"
May 26 17:17:37.556737 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.556713    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-bb656698-djxbk"]
May 26 17:17:37.562553 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.562535    2571 scope.go:117] "RemoveContainer" containerID="3d6868df34461ff1b5372604f0068ba072f6c949150a576d2227b339c927a3ca"
May 26 17:17:37.569523 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.569506    2571 scope.go:117] "RemoveContainer" containerID="b8abe972bff55bf53fe181659cf83454a383c29329388fc1b529dff16e3a3103"
May 26 17:17:37.834847 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.834761    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" path="/var/lib/kubelet/pods/6aaef280-cff6-4b47-8999-4f90926ec19c/volumes"
May 26 17:17:37.835341 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:37.835327    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed31bff7-927f-4978-967a-1dcf00328330" path="/var/lib/kubelet/pods/ed31bff7-927f-4978-967a-1dcf00328330/volumes"
May 26 17:17:48.897588 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897509    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-m97fp"]
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897850    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="feast-init"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897862    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="feast-init"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897874    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="feast-apply"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897882    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="feast-apply"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897895    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="feast-init"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897904    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="feast-init"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897913    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="feast-apply"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897920    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="feast-apply"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897927    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="offline"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897935    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="offline"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897945    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="registry"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897952    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="registry"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897960    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="ui"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897968    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="ui"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897983    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="offline"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.897990    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="offline"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898026    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="ui"
May 26 17:17:48.898025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898038    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="ui"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898048    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="online"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898056    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="online"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898067    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="online"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898074    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="online"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898161    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="ui"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898175    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="offline"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898186    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="online"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898200    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="ui"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898209    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="offline"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898221    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed31bff7-927f-4978-967a-1dcf00328330" containerName="registry"
May 26 17:17:48.898719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.898229    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="6aaef280-cff6-4b47-8999-4f90926ec19c" containerName="online"
May 26 17:17:48.901233 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.901208    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:17:48.904059 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.904035    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 26 17:17:48.904185 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.904040    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 26 17:17:48.905179 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.905157    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 26 17:17:48.905443 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.905215    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-zr72g\""
May 26 17:17:48.907189 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:48.907169    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-m97fp"]
May 26 17:17:49.040903 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:49.040863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2pj7\" (UniqueName: \"kubernetes.io/projected/59deaf3a-2124-47e7-9801-096eaf04def7-kube-api-access-j2pj7\") pod \"postgres-7c7656cc6d-m97fp\" (UID: \"59deaf3a-2124-47e7-9801-096eaf04def7\") " pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:17:49.041098 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:49.040925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/59deaf3a-2124-47e7-9801-096eaf04def7-postgresdata\") pod \"postgres-7c7656cc6d-m97fp\" (UID: \"59deaf3a-2124-47e7-9801-096eaf04def7\") " pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:17:49.142372 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:49.142336    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j2pj7\" (UniqueName: \"kubernetes.io/projected/59deaf3a-2124-47e7-9801-096eaf04def7-kube-api-access-j2pj7\") pod \"postgres-7c7656cc6d-m97fp\" (UID: \"59deaf3a-2124-47e7-9801-096eaf04def7\") " pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:17:49.142533 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:49.142391    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/59deaf3a-2124-47e7-9801-096eaf04def7-postgresdata\") pod \"postgres-7c7656cc6d-m97fp\" (UID: \"59deaf3a-2124-47e7-9801-096eaf04def7\") " pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:17:49.142740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:49.142723    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/59deaf3a-2124-47e7-9801-096eaf04def7-postgresdata\") pod \"postgres-7c7656cc6d-m97fp\" (UID: \"59deaf3a-2124-47e7-9801-096eaf04def7\") " pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:17:49.150324 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:49.150255    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2pj7\" (UniqueName: \"kubernetes.io/projected/59deaf3a-2124-47e7-9801-096eaf04def7-kube-api-access-j2pj7\") pod \"postgres-7c7656cc6d-m97fp\" (UID: \"59deaf3a-2124-47e7-9801-096eaf04def7\") " pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:17:49.221579 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:49.221531    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:17:49.346409 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:49.346378    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-m97fp"]
May 26 17:17:49.349559 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:17:49.349531    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59deaf3a_2124_47e7_9801_096eaf04def7.slice/crio-969ad52cb5456bbf80b320c693f711465de193fdd299d82e4baafc0c050d926b WatchSource:0}: Error finding container 969ad52cb5456bbf80b320c693f711465de193fdd299d82e4baafc0c050d926b: Status 404 returned error can't find the container with id 969ad52cb5456bbf80b320c693f711465de193fdd299d82e4baafc0c050d926b
May 26 17:17:49.574400 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:49.574364    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-m97fp" event={"ID":"59deaf3a-2124-47e7-9801-096eaf04def7","Type":"ContainerStarted","Data":"969ad52cb5456bbf80b320c693f711465de193fdd299d82e4baafc0c050d926b"}
May 26 17:17:55.603202 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:55.603158    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-m97fp" event={"ID":"59deaf3a-2124-47e7-9801-096eaf04def7","Type":"ContainerStarted","Data":"a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d"}
May 26 17:17:55.618276 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:55.618228    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-m97fp" podStartSLOduration=1.528536796 podStartE2EDuration="7.618212995s" podCreationTimestamp="2026-05-26 17:17:48 +0000 UTC" firstStartedPulling="2026-05-26 17:17:49.351258156 +0000 UTC m=+810.076347638" lastFinishedPulling="2026-05-26 17:17:55.440934359 +0000 UTC m=+816.166023837" observedRunningTime="2026-05-26 17:17:55.61626711 +0000 UTC m=+816.341356612" watchObservedRunningTime="2026-05-26 17:17:55.618212995 +0000 UTC m=+816.343302496"
May 26 17:17:59.616482 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.616443    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"]
May 26 17:17:59.620702 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.620681    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.623419 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.623389    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 26 17:17:59.623548 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.623423    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 26 17:17:59.623548 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.623493    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 26 17:17:59.623667 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.623549    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-f6jm2\""
May 26 17:17:59.623760 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.623742    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 26 17:17:59.629413 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.629391    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"]
May 26 17:17:59.738175 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.738142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-registry-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.738329 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.738181    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-online-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.738329 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.738276    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/908652e4-3c63-495f-b853-bba0754d52a2-feast-data\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.738329 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.738310    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2tbb\" (UniqueName: \"kubernetes.io/projected/908652e4-3c63-495f-b853-bba0754d52a2-kube-api-access-p2tbb\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.738453 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.738337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-offline-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.838772 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.838738    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/908652e4-3c63-495f-b853-bba0754d52a2-feast-data\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.838772 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.838777    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p2tbb\" (UniqueName: \"kubernetes.io/projected/908652e4-3c63-495f-b853-bba0754d52a2-kube-api-access-p2tbb\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.838989 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.838797    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-offline-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.838989 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.838833    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-registry-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.838989 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.838854    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-online-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.838989 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:17:59.838969    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found
May 26 17:17:59.839205 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:17:59.839086    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-registry-tls podName:908652e4-3c63-495f-b853-bba0754d52a2 nodeName:}" failed. No retries permitted until 2026-05-26 17:18:00.339064978 +0000 UTC m=+821.064154462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-registry-tls") pod "feast-credit-scoring-79db59fb8b-vxzz4" (UID: "908652e4-3c63-495f-b853-bba0754d52a2") : secret "feast-credit-scoring-registry-tls" not found
May 26 17:17:59.839205 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.839181    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/908652e4-3c63-495f-b853-bba0754d52a2-feast-data\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.841354 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.841322    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-offline-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.841465 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.841417    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-online-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:17:59.848903 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:17:59.848875    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2tbb\" (UniqueName: \"kubernetes.io/projected/908652e4-3c63-495f-b853-bba0754d52a2-kube-api-access-p2tbb\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:00.344050 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:00.343991    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-registry-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:00.346593 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:00.346564    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-registry-tls\") pod \"feast-credit-scoring-79db59fb8b-vxzz4\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") " pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:00.531363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:00.531325    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:00.668739 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:00.668712    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"]
May 26 17:18:00.669792 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:18:00.669766    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908652e4_3c63_495f_b853_bba0754d52a2.slice/crio-e13d66814c454960e6f32ee99f60cbbf37b3107ebd76e85aae946bbde89363ab WatchSource:0}: Error finding container e13d66814c454960e6f32ee99f60cbbf37b3107ebd76e85aae946bbde89363ab: Status 404 returned error can't find the container with id e13d66814c454960e6f32ee99f60cbbf37b3107ebd76e85aae946bbde89363ab
May 26 17:18:01.625860 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:01.625828    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerStarted","Data":"4c786e0bd5a36e6e1d7c05ac4d33291a9de81d4d4057bf5b3bf776702d9795db"}
May 26 17:18:01.625860 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:01.625864    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerStarted","Data":"e13d66814c454960e6f32ee99f60cbbf37b3107ebd76e85aae946bbde89363ab"}
May 26 17:18:02.630058 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:02.630005    2571 generic.go:358] "Generic (PLEG): container finished" podID="908652e4-3c63-495f-b853-bba0754d52a2" containerID="4c786e0bd5a36e6e1d7c05ac4d33291a9de81d4d4057bf5b3bf776702d9795db" exitCode=0
May 26 17:18:02.630058 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:02.630051    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerDied","Data":"4c786e0bd5a36e6e1d7c05ac4d33291a9de81d4d4057bf5b3bf776702d9795db"}
May 26 17:18:03.635417 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:03.635377    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerStarted","Data":"3f6527d7b2c9fde42c3839936c80d403569eb572338ad8807d25eba66e9191e9"}
May 26 17:18:09.656994 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:09.656959    2571 generic.go:358] "Generic (PLEG): container finished" podID="908652e4-3c63-495f-b853-bba0754d52a2" containerID="3f6527d7b2c9fde42c3839936c80d403569eb572338ad8807d25eba66e9191e9" exitCode=0
May 26 17:18:09.657443 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:09.657047    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerDied","Data":"3f6527d7b2c9fde42c3839936c80d403569eb572338ad8807d25eba66e9191e9"}
May 26 17:18:10.663562 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:10.663525    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerStarted","Data":"d8880abfcae6222177f3313779348b3cf5106598bb692a29d4d005f7149a804a"}
May 26 17:18:10.663562 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:10.663564    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerStarted","Data":"a1bf698a8beb1a8d3f60e0bf5972a52aceaeb4964dba822cf0db16d2afadb689"}
May 26 17:18:10.664135 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:10.663574    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerStarted","Data":"e7871e79d9f221ae0b037cd8b575494e99ff9eb0539fa06d1ee51546dd5f11c6"}
May 26 17:18:10.688739 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:10.688682    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podStartSLOduration=11.688663503 podStartE2EDuration="11.688663503s" podCreationTimestamp="2026-05-26 17:17:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:18:10.687189705 +0000 UTC m=+831.412279207" watchObservedRunningTime="2026-05-26 17:18:10.688663503 +0000 UTC m=+831.413753005"
May 26 17:18:12.532195 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:12.532148    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:12.532195 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:12.532206    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:12.532738 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:12.532222    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:12.533941 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:12.533901    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused"
May 26 17:18:12.534093 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:12.533942    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused"
May 26 17:18:12.534093 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:12.533944    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 26 17:18:15.532385 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:15.532338    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused"
May 26 17:18:15.533299 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:15.533268    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:15.533922 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:15.533903    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:15.534319 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:15.534301    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:15.877326 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:15.877244    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:16.696853 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:16.696817    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:16.700956 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:16.700930    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:18.532118 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:18.532070    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused"
May 26 17:18:20.532478 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:20.532435    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:21.532997 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:21.532960    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:18:21.533703 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:18:21.533684    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:19:19.786888 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:19.786855    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:19:19.787465 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:19.787136    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:19:25.478546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.478503    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"]
May 26 17:19:25.479138 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.479103    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="offline" containerID="cri-o://d8880abfcae6222177f3313779348b3cf5106598bb692a29d4d005f7149a804a" gracePeriod=30
May 26 17:19:25.479274 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.479248    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="registry" containerID="cri-o://e7871e79d9f221ae0b037cd8b575494e99ff9eb0539fa06d1ee51546dd5f11c6" gracePeriod=30
May 26 17:19:25.479350 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.479226    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="online" containerID="cri-o://a1bf698a8beb1a8d3f60e0bf5972a52aceaeb4964dba822cf0db16d2afadb689" gracePeriod=30
May 26 17:19:25.493558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.493531    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-m97fp"]
May 26 17:19:25.493725 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.493705    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-m97fp" podUID="59deaf3a-2124-47e7-9801-096eaf04def7" containerName="postgres" containerID="cri-o://a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d" gracePeriod=30
May 26 17:19:25.739354 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.739332    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:19:25.781650 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.781610    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/59deaf3a-2124-47e7-9801-096eaf04def7-postgresdata\") pod \"59deaf3a-2124-47e7-9801-096eaf04def7\" (UID: \"59deaf3a-2124-47e7-9801-096eaf04def7\") "
May 26 17:19:25.781813 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.781692    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2pj7\" (UniqueName: \"kubernetes.io/projected/59deaf3a-2124-47e7-9801-096eaf04def7-kube-api-access-j2pj7\") pod \"59deaf3a-2124-47e7-9801-096eaf04def7\" (UID: \"59deaf3a-2124-47e7-9801-096eaf04def7\") "
May 26 17:19:25.784149 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.783981    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59deaf3a-2124-47e7-9801-096eaf04def7-kube-api-access-j2pj7" (OuterVolumeSpecName: "kube-api-access-j2pj7") pod "59deaf3a-2124-47e7-9801-096eaf04def7" (UID: "59deaf3a-2124-47e7-9801-096eaf04def7"). InnerVolumeSpecName "kube-api-access-j2pj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:19:25.812979 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.812939    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59deaf3a-2124-47e7-9801-096eaf04def7-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "59deaf3a-2124-47e7-9801-096eaf04def7" (UID: "59deaf3a-2124-47e7-9801-096eaf04def7"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:19:25.883046 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.882984    2571 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/59deaf3a-2124-47e7-9801-096eaf04def7-postgresdata\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:19:25.883046 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.883038    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2pj7\" (UniqueName: \"kubernetes.io/projected/59deaf3a-2124-47e7-9801-096eaf04def7-kube-api-access-j2pj7\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:19:25.934075 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.934001    2571 generic.go:358] "Generic (PLEG): container finished" podID="59deaf3a-2124-47e7-9801-096eaf04def7" containerID="a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d" exitCode=0
May 26 17:19:25.934075 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.934066    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-m97fp"
May 26 17:19:25.934295 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.934087    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-m97fp" event={"ID":"59deaf3a-2124-47e7-9801-096eaf04def7","Type":"ContainerDied","Data":"a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d"}
May 26 17:19:25.934295 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.934131    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-m97fp" event={"ID":"59deaf3a-2124-47e7-9801-096eaf04def7","Type":"ContainerDied","Data":"969ad52cb5456bbf80b320c693f711465de193fdd299d82e4baafc0c050d926b"}
May 26 17:19:25.934295 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.934154    2571 scope.go:117] "RemoveContainer" containerID="a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d"
May 26 17:19:25.944250 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.944231    2571 scope.go:117] "RemoveContainer" containerID="a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d"
May 26 17:19:25.944527 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:19:25.944493    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d\": container with ID starting with a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d not found: ID does not exist" containerID="a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d"
May 26 17:19:25.944611 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.944523    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d"} err="failed to get container status \"a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d\": rpc error: code = NotFound desc = could not find container \"a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d\": container with ID starting with a107d7250ff98f5a29d4b6e428bae8942158cbbd6f082db4b0d363ee2521e38d not found: ID does not exist"
May 26 17:19:25.952616 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.952591    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-m97fp"]
May 26 17:19:25.956200 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:25.956180    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-m97fp"]
May 26 17:19:26.697778 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:26.697717    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 26 17:19:26.940619 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:26.940589    2571 generic.go:358] "Generic (PLEG): container finished" podID="908652e4-3c63-495f-b853-bba0754d52a2" containerID="a1bf698a8beb1a8d3f60e0bf5972a52aceaeb4964dba822cf0db16d2afadb689" exitCode=0
May 26 17:19:26.940787 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:26.940629    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerDied","Data":"a1bf698a8beb1a8d3f60e0bf5972a52aceaeb4964dba822cf0db16d2afadb689"}
May 26 17:19:27.834875 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:27.834839    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59deaf3a-2124-47e7-9801-096eaf04def7" path="/var/lib/kubelet/pods/59deaf3a-2124-47e7-9801-096eaf04def7/volumes"
May 26 17:19:36.697506 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:36.697461    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 26 17:19:46.697332 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:46.697290    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 26 17:19:46.698806 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:46.697427    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:19:55.534613 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:55.534578    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused"
May 26 17:19:56.049888 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.049851    2571 generic.go:358] "Generic (PLEG): container finished" podID="908652e4-3c63-495f-b853-bba0754d52a2" containerID="d8880abfcae6222177f3313779348b3cf5106598bb692a29d4d005f7149a804a" exitCode=137
May 26 17:19:56.049888 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.049884    2571 generic.go:358] "Generic (PLEG): container finished" podID="908652e4-3c63-495f-b853-bba0754d52a2" containerID="e7871e79d9f221ae0b037cd8b575494e99ff9eb0539fa06d1ee51546dd5f11c6" exitCode=137
May 26 17:19:56.050111 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.049927    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerDied","Data":"d8880abfcae6222177f3313779348b3cf5106598bb692a29d4d005f7149a804a"}
May 26 17:19:56.050111 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.049982    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerDied","Data":"e7871e79d9f221ae0b037cd8b575494e99ff9eb0539fa06d1ee51546dd5f11c6"}
May 26 17:19:56.134071 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.134047    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:19:56.250145 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.249989    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-registry-tls\") pod \"908652e4-3c63-495f-b853-bba0754d52a2\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") "
May 26 17:19:56.250145 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.250088    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-online-tls\") pod \"908652e4-3c63-495f-b853-bba0754d52a2\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") "
May 26 17:19:56.250145 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.250145    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-offline-tls\") pod \"908652e4-3c63-495f-b853-bba0754d52a2\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") "
May 26 17:19:56.250440 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.250183    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/908652e4-3c63-495f-b853-bba0754d52a2-feast-data\") pod \"908652e4-3c63-495f-b853-bba0754d52a2\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") "
May 26 17:19:56.250440 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.250207    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2tbb\" (UniqueName: \"kubernetes.io/projected/908652e4-3c63-495f-b853-bba0754d52a2-kube-api-access-p2tbb\") pod \"908652e4-3c63-495f-b853-bba0754d52a2\" (UID: \"908652e4-3c63-495f-b853-bba0754d52a2\") "
May 26 17:19:56.252225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.252189    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "908652e4-3c63-495f-b853-bba0754d52a2" (UID: "908652e4-3c63-495f-b853-bba0754d52a2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:19:56.252356 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.252227    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-online-tls" (OuterVolumeSpecName: "online-tls") pod "908652e4-3c63-495f-b853-bba0754d52a2" (UID: "908652e4-3c63-495f-b853-bba0754d52a2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:19:56.252356 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.252271    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "908652e4-3c63-495f-b853-bba0754d52a2" (UID: "908652e4-3c63-495f-b853-bba0754d52a2"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:19:56.252592 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.252578    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908652e4-3c63-495f-b853-bba0754d52a2-kube-api-access-p2tbb" (OuterVolumeSpecName: "kube-api-access-p2tbb") pod "908652e4-3c63-495f-b853-bba0754d52a2" (UID: "908652e4-3c63-495f-b853-bba0754d52a2"). InnerVolumeSpecName "kube-api-access-p2tbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:19:56.258735 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.258705    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908652e4-3c63-495f-b853-bba0754d52a2-feast-data" (OuterVolumeSpecName: "feast-data") pod "908652e4-3c63-495f-b853-bba0754d52a2" (UID: "908652e4-3c63-495f-b853-bba0754d52a2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:19:56.351407 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.351366    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/908652e4-3c63-495f-b853-bba0754d52a2-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:19:56.351407 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.351399    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2tbb\" (UniqueName: \"kubernetes.io/projected/908652e4-3c63-495f-b853-bba0754d52a2-kube-api-access-p2tbb\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:19:56.351407 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.351412    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:19:56.351407 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.351421    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:19:56.351670 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:56.351429    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/908652e4-3c63-495f-b853-bba0754d52a2-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:19:57.056297 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.056263    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4" event={"ID":"908652e4-3c63-495f-b853-bba0754d52a2","Type":"ContainerDied","Data":"e13d66814c454960e6f32ee99f60cbbf37b3107ebd76e85aae946bbde89363ab"}
May 26 17:19:57.056781 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.056310    2571 scope.go:117] "RemoveContainer" containerID="d8880abfcae6222177f3313779348b3cf5106598bb692a29d4d005f7149a804a"
May 26 17:19:57.056781 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.056272    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"
May 26 17:19:57.065284 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.065266    2571 scope.go:117] "RemoveContainer" containerID="a1bf698a8beb1a8d3f60e0bf5972a52aceaeb4964dba822cf0db16d2afadb689"
May 26 17:19:57.073540 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.073516    2571 scope.go:117] "RemoveContainer" containerID="e7871e79d9f221ae0b037cd8b575494e99ff9eb0539fa06d1ee51546dd5f11c6"
May 26 17:19:57.080156 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.080131    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"]
May 26 17:19:57.082789 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.082772    2571 scope.go:117] "RemoveContainer" containerID="3f6527d7b2c9fde42c3839936c80d403569eb572338ad8807d25eba66e9191e9"
May 26 17:19:57.085892 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.085866    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-79db59fb8b-vxzz4"]
May 26 17:19:57.091607 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.091586    2571 scope.go:117] "RemoveContainer" containerID="4c786e0bd5a36e6e1d7c05ac4d33291a9de81d4d4057bf5b3bf776702d9795db"
May 26 17:19:57.835062 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:19:57.834998    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908652e4-3c63-495f-b853-bba0754d52a2" path="/var/lib/kubelet/pods/908652e4-3c63-495f-b853-bba0754d52a2/volumes"
May 26 17:21:26.929546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.929515    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-r5drb"]
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.929917    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="online"
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.929934    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="online"
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.929951    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="feast-init"
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.929960    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="feast-init"
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.929976    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="registry"
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.929982    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="registry"
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.929993    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="feast-apply"
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.929999    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="feast-apply"
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.930007    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59deaf3a-2124-47e7-9801-096eaf04def7" containerName="postgres"
May 26 17:21:26.930031 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.930030    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="59deaf3a-2124-47e7-9801-096eaf04def7" containerName="postgres"
May 26 17:21:26.930374 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.930044    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="offline"
May 26 17:21:26.930374 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.930049    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="offline"
May 26 17:21:26.930374 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.930115    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="59deaf3a-2124-47e7-9801-096eaf04def7" containerName="postgres"
May 26 17:21:26.930374 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.930124    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="registry"
May 26 17:21:26.930374 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.930130    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="online"
May 26 17:21:26.930374 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.930137    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="908652e4-3c63-495f-b853-bba0754d52a2" containerName="offline"
May 26 17:21:26.932295 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.932272    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-r5drb"
May 26 17:21:26.934842 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.934823    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 26 17:21:26.934973 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.934853    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-hbcc6\""
May 26 17:21:26.936122 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.935983    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 26 17:21:26.946870 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.946845    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-r5drb"]
May 26 17:21:26.997220 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:26.997185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2dl\" (UniqueName: \"kubernetes.io/projected/d211cd58-7973-4a26-ade1-b9b3e5e91416-kube-api-access-fp2dl\") pod \"redis-854544c579-r5drb\" (UID: \"d211cd58-7973-4a26-ade1-b9b3e5e91416\") " pod="test-ns-feast-rest/redis-854544c579-r5drb"
May 26 17:21:27.098531 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:27.098493    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2dl\" (UniqueName: \"kubernetes.io/projected/d211cd58-7973-4a26-ade1-b9b3e5e91416-kube-api-access-fp2dl\") pod \"redis-854544c579-r5drb\" (UID: \"d211cd58-7973-4a26-ade1-b9b3e5e91416\") " pod="test-ns-feast-rest/redis-854544c579-r5drb"
May 26 17:21:27.107221 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:27.107190    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2dl\" (UniqueName: \"kubernetes.io/projected/d211cd58-7973-4a26-ade1-b9b3e5e91416-kube-api-access-fp2dl\") pod \"redis-854544c579-r5drb\" (UID: \"d211cd58-7973-4a26-ade1-b9b3e5e91416\") " pod="test-ns-feast-rest/redis-854544c579-r5drb"
May 26 17:21:27.249970 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:27.249937    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-r5drb"
May 26 17:21:27.379240 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:27.379068    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-r5drb"]
May 26 17:21:27.381607 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:21:27.381576    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd211cd58_7973_4a26_ade1_b9b3e5e91416.slice/crio-a76c98375747ad9dfba7471a69cab7ed5bf9bbafe1e778b54e48af92252848f2 WatchSource:0}: Error finding container a76c98375747ad9dfba7471a69cab7ed5bf9bbafe1e778b54e48af92252848f2: Status 404 returned error can't find the container with id a76c98375747ad9dfba7471a69cab7ed5bf9bbafe1e778b54e48af92252848f2
May 26 17:21:27.383191 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:27.383175    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 17:21:28.341185 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:28.341142    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-r5drb" event={"ID":"d211cd58-7973-4a26-ade1-b9b3e5e91416","Type":"ContainerStarted","Data":"a76c98375747ad9dfba7471a69cab7ed5bf9bbafe1e778b54e48af92252848f2"}
May 26 17:21:29.346205 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:29.346170    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-r5drb" event={"ID":"d211cd58-7973-4a26-ade1-b9b3e5e91416","Type":"ContainerStarted","Data":"bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba"}
May 26 17:21:29.361400 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:29.361338    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-r5drb" podStartSLOduration=1.991525453 podStartE2EDuration="3.361324691s" podCreationTimestamp="2026-05-26 17:21:26 +0000 UTC" firstStartedPulling="2026-05-26 17:21:27.383331399 +0000 UTC m=+1028.108420878" lastFinishedPulling="2026-05-26 17:21:28.753130633 +0000 UTC m=+1029.478220116" observedRunningTime="2026-05-26 17:21:29.359386494 +0000 UTC m=+1030.084475996" watchObservedRunningTime="2026-05-26 17:21:29.361324691 +0000 UTC m=+1030.086414191"
May 26 17:21:32.348563 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.348518    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-mbth5"]
May 26 17:21:32.351573 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.351548    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:21:32.354567 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.354543    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 26 17:21:32.360756 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.360290    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-mbth5"]
May 26 17:21:32.447917 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.447878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjq7\" (UniqueName: \"kubernetes.io/projected/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-kube-api-access-mqjq7\") pod \"postgres-7c7656cc6d-mbth5\" (UID: \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:21:32.448130 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.447953    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-postgresdata\") pod \"postgres-7c7656cc6d-mbth5\" (UID: \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:21:32.548824 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.548790    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-postgresdata\") pod \"postgres-7c7656cc6d-mbth5\" (UID: \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:21:32.548984 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.548936    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjq7\" (UniqueName: \"kubernetes.io/projected/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-kube-api-access-mqjq7\") pod \"postgres-7c7656cc6d-mbth5\" (UID: \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:21:32.549158 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.549138    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-postgresdata\") pod \"postgres-7c7656cc6d-mbth5\" (UID: \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:21:32.557363 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.557330    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjq7\" (UniqueName: \"kubernetes.io/projected/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-kube-api-access-mqjq7\") pod \"postgres-7c7656cc6d-mbth5\" (UID: \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:21:32.664241 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.664160    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:21:32.792520 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:32.792437    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-mbth5"]
May 26 17:21:32.795159 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:21:32.795124    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod463b5edc_b6d9_4f5a_a2b0_80cf86b30575.slice/crio-cc04bd301382dceb76c33b9b44ada7d4aea346230a17bc2f722fed47bf2124d2 WatchSource:0}: Error finding container cc04bd301382dceb76c33b9b44ada7d4aea346230a17bc2f722fed47bf2124d2: Status 404 returned error can't find the container with id cc04bd301382dceb76c33b9b44ada7d4aea346230a17bc2f722fed47bf2124d2
May 26 17:21:33.361164 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:33.361126    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5" event={"ID":"463b5edc-b6d9-4f5a-a2b0-80cf86b30575","Type":"ContainerStarted","Data":"b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3"}
May 26 17:21:33.361164 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:33.361168    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5" event={"ID":"463b5edc-b6d9-4f5a-a2b0-80cf86b30575","Type":"ContainerStarted","Data":"cc04bd301382dceb76c33b9b44ada7d4aea346230a17bc2f722fed47bf2124d2"}
May 26 17:21:33.378469 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:33.378422    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5" podStartSLOduration=1.378408509 podStartE2EDuration="1.378408509s" podCreationTimestamp="2026-05-26 17:21:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:21:33.377208442 +0000 UTC m=+1034.102297958" watchObservedRunningTime="2026-05-26 17:21:33.378408509 +0000 UTC m=+1034.103498010"
May 26 17:21:37.978490 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:37.978455    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"]
May 26 17:21:37.985242 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:37.985216    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:37.987989 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:37.987934    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 26 17:21:37.988340 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:37.987933    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 26 17:21:37.988571 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:37.987957    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 26 17:21:37.988730 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:37.987988    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-nttrs\""
May 26 17:21:37.990893 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:37.990872    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"]
May 26 17:21:38.096417 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.096380    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-registry-tls\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.096417 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.096420    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-online-tls\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.096648 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.096499    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc138561-06d2-4d1f-8ace-9b33a7146ea3-feast-data\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.096648 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.096543    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxkf\" (UniqueName: \"kubernetes.io/projected/dc138561-06d2-4d1f-8ace-9b33a7146ea3-kube-api-access-xpxkf\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.197460 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.197422    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc138561-06d2-4d1f-8ace-9b33a7146ea3-feast-data\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.197460 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.197466    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxkf\" (UniqueName: \"kubernetes.io/projected/dc138561-06d2-4d1f-8ace-9b33a7146ea3-kube-api-access-xpxkf\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.197695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.197515    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-registry-tls\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.197695 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.197533    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-online-tls\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.197695 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:21:38.197668    2571 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found
May 26 17:21:38.197841 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:21:38.197710    2571 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found
May 26 17:21:38.197841 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:21:38.197744    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-registry-tls podName:dc138561-06d2-4d1f-8ace-9b33a7146ea3 nodeName:}" failed. No retries permitted until 2026-05-26 17:21:38.697721902 +0000 UTC m=+1039.422811396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-registry-tls") pod "feast-credit-scoring-5847df4f87-qnl8n" (UID: "dc138561-06d2-4d1f-8ace-9b33a7146ea3") : secret "feast-credit-scoring-registry-tls" not found
May 26 17:21:38.197841 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:21:38.197765    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-online-tls podName:dc138561-06d2-4d1f-8ace-9b33a7146ea3 nodeName:}" failed. No retries permitted until 2026-05-26 17:21:38.697755611 +0000 UTC m=+1039.422845096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-online-tls") pod "feast-credit-scoring-5847df4f87-qnl8n" (UID: "dc138561-06d2-4d1f-8ace-9b33a7146ea3") : secret "feast-credit-scoring-online-tls" not found
May 26 17:21:38.197947 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.197856    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc138561-06d2-4d1f-8ace-9b33a7146ea3-feast-data\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.208993 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.208965    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxkf\" (UniqueName: \"kubernetes.io/projected/dc138561-06d2-4d1f-8ace-9b33a7146ea3-kube-api-access-xpxkf\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.706434 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.706397    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-registry-tls\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.706434 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.706439    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-online-tls\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.708980 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.708952    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-registry-tls\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.709145 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.709003    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-online-tls\") pod \"feast-credit-scoring-5847df4f87-qnl8n\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") " pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:38.896116 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:38.896075    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:39.025055 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:39.025002    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"]
May 26 17:21:39.027760 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:21:39.027732    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc138561_06d2_4d1f_8ace_9b33a7146ea3.slice/crio-5efae3795d3f0faac869f2d2f1a7cdcb20bfc70605c4aba5f94243149581f1d3 WatchSource:0}: Error finding container 5efae3795d3f0faac869f2d2f1a7cdcb20bfc70605c4aba5f94243149581f1d3: Status 404 returned error can't find the container with id 5efae3795d3f0faac869f2d2f1a7cdcb20bfc70605c4aba5f94243149581f1d3
May 26 17:21:39.384095 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:39.384059    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerStarted","Data":"8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0"}
May 26 17:21:39.384095 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:39.384098    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerStarted","Data":"5efae3795d3f0faac869f2d2f1a7cdcb20bfc70605c4aba5f94243149581f1d3"}
May 26 17:21:40.388764 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:40.388673    2571 generic.go:358] "Generic (PLEG): container finished" podID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerID="8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0" exitCode=0
May 26 17:21:40.388764 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:40.388717    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerDied","Data":"8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0"}
May 26 17:21:41.394127 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:41.394091    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerStarted","Data":"e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7"}
May 26 17:21:48.420226 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:48.420194    2571 generic.go:358] "Generic (PLEG): container finished" podID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerID="e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7" exitCode=0
May 26 17:21:48.420679 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:48.420275    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerDied","Data":"e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7"}
May 26 17:21:49.431067 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:49.431005    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerStarted","Data":"abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310"}
May 26 17:21:49.431067 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:49.431069    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerStarted","Data":"30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed"}
May 26 17:21:49.453875 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:49.453813    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podStartSLOduration=12.453798873 podStartE2EDuration="12.453798873s" podCreationTimestamp="2026-05-26 17:21:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:21:49.451456896 +0000 UTC m=+1050.176546402" watchObservedRunningTime="2026-05-26 17:21:49.453798873 +0000 UTC m=+1050.178888373"
May 26 17:21:50.896816 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:50.896771    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:50.897309 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:50.896832    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:50.898740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:50.898704    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused"
May 26 17:21:50.898740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:50.898722    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 26 17:21:53.897291 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:53.897248    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused"
May 26 17:21:53.897781 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:53.897345    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 26 17:21:56.896555 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:56.896507    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused"
May 26 17:21:56.901854 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:56.901827    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:56.902373 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:56.902351    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:56.905236 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:56.905212    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:58.896779 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:58.896740    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:21:59.897506 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:21:59.897476    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:22:00.472054 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:00.471999    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:22:03.809301 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.809267    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"]
May 26 17:22:03.814080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.814059    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.816657 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.816630    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 26 17:22:03.816826 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.816685    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-z8tjc\""
May 26 17:22:03.816826 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.816688    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 26 17:22:03.816971 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.816950    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 26 17:22:03.823396 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.823373    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"]
May 26 17:22:03.837170 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.837142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-online-tls\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.837329 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.837186    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-registry-tls\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.837329 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.837267    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4516b210-ae0f-4c57-8980-e56081f106f9-feast-data\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.837329 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.837321    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rgz\" (UniqueName: \"kubernetes.io/projected/4516b210-ae0f-4c57-8980-e56081f106f9-kube-api-access-w6rgz\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.938097 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.938061    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-registry-tls\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.938097 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.938104    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4516b210-ae0f-4c57-8980-e56081f106f9-feast-data\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.938345 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.938151    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rgz\" (UniqueName: \"kubernetes.io/projected/4516b210-ae0f-4c57-8980-e56081f106f9-kube-api-access-w6rgz\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.938345 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.938255    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-online-tls\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.938596 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.938568    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4516b210-ae0f-4c57-8980-e56081f106f9-feast-data\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.940634 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.940607    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-registry-tls\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.940840 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.940818    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-online-tls\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:03.946128 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:03.946107    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rgz\" (UniqueName: \"kubernetes.io/projected/4516b210-ae0f-4c57-8980-e56081f106f9-kube-api-access-w6rgz\") pod \"feast-driver-ranking-6f5794f64d-6rkkz\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") " pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:04.125497 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:04.125402    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:04.259258 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:04.259234    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"]
May 26 17:22:04.261995 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:22:04.261965    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4516b210_ae0f_4c57_8980_e56081f106f9.slice/crio-7ef1f01c0b466bc33912e6a64e4cb668e103286dca6c1cbf62ea5ac0ba17d41e WatchSource:0}: Error finding container 7ef1f01c0b466bc33912e6a64e4cb668e103286dca6c1cbf62ea5ac0ba17d41e: Status 404 returned error can't find the container with id 7ef1f01c0b466bc33912e6a64e4cb668e103286dca6c1cbf62ea5ac0ba17d41e
May 26 17:22:04.485577 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:04.485484    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerStarted","Data":"f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858"}
May 26 17:22:04.485577 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:04.485530    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerStarted","Data":"7ef1f01c0b466bc33912e6a64e4cb668e103286dca6c1cbf62ea5ac0ba17d41e"}
May 26 17:22:09.503121 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:09.503088    2571 generic.go:358] "Generic (PLEG): container finished" podID="4516b210-ae0f-4c57-8980-e56081f106f9" containerID="f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858" exitCode=0
May 26 17:22:09.503513 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:09.503171    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerDied","Data":"f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858"}
May 26 17:22:10.508759 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:10.508717    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerStarted","Data":"300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c"}
May 26 17:22:17.534530 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:17.534494    2571 generic.go:358] "Generic (PLEG): container finished" podID="4516b210-ae0f-4c57-8980-e56081f106f9" containerID="300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c" exitCode=0
May 26 17:22:17.534920 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:17.534564    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerDied","Data":"300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c"}
May 26 17:22:18.540983 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:18.540948    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerStarted","Data":"b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d"}
May 26 17:22:18.540983 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:18.540985    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerStarted","Data":"7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6"}
May 26 17:22:18.563375 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:18.563311    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podStartSLOduration=15.563290539 podStartE2EDuration="15.563290539s" podCreationTimestamp="2026-05-26 17:22:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:22:18.560880054 +0000 UTC m=+1079.285969555" watchObservedRunningTime="2026-05-26 17:22:18.563290539 +0000 UTC m=+1079.288380042"
May 26 17:22:19.126096 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:19.126052    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:19.126311 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:19.126113    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:19.127663 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:19.127627    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.37:6571: connect: connection refused"
May 26 17:22:19.127824 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:19.127677    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused"
May 26 17:22:22.126816 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:22.126771    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused"
May 26 17:22:22.127233 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:22.126773    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.37:6571: connect: connection refused"
May 26 17:22:24.125772 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:24.125729    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:24.125772 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:24.125772    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:25.126222 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:25.126166    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.37:6571: connect: connection refused"
May 26 17:22:25.131386 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:25.131359    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:25.135760 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:25.135735    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:28.126398 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:28.126339    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.37:6571: connect: connection refused"
May 26 17:22:31.126745 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:31.126718    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:22:31.127446 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:22:31.127425    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:23:47.998712 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:47.998632    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"]
May 26 17:23:47.999216 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:47.998904    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg" podUID="2a13712f-2bef-4ee2-bc14-21157355d517" containerName="manager" containerID="cri-o://605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4" gracePeriod=10
May 26 17:23:48.268897 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.268860    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"]
May 26 17:23:48.269252 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.269221    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="registry" containerID="cri-o://30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed" gracePeriod=30
May 26 17:23:48.269252 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.269247    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="online" containerID="cri-o://abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310" gracePeriod=30
May 26 17:23:48.272764 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.272738    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"]
May 26 17:23:48.273238 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.273166    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="registry" containerID="cri-o://7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6" gracePeriod=30
May 26 17:23:48.273462 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.273413    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="online" containerID="cri-o://b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d" gracePeriod=30
May 26 17:23:48.291069 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.291041    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"
May 26 17:23:48.342491 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.342461    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwf9n\" (UniqueName: \"kubernetes.io/projected/2a13712f-2bef-4ee2-bc14-21157355d517-kube-api-access-lwf9n\") pod \"2a13712f-2bef-4ee2-bc14-21157355d517\" (UID: \"2a13712f-2bef-4ee2-bc14-21157355d517\") "
May 26 17:23:48.344680 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.344651    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a13712f-2bef-4ee2-bc14-21157355d517-kube-api-access-lwf9n" (OuterVolumeSpecName: "kube-api-access-lwf9n") pod "2a13712f-2bef-4ee2-bc14-21157355d517" (UID: "2a13712f-2bef-4ee2-bc14-21157355d517"). InnerVolumeSpecName "kube-api-access-lwf9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:23:48.443671 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.443635    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwf9n\" (UniqueName: \"kubernetes.io/projected/2a13712f-2bef-4ee2-bc14-21157355d517-kube-api-access-lwf9n\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:23:48.865539 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.865499    2571 generic.go:358] "Generic (PLEG): container finished" podID="2a13712f-2bef-4ee2-bc14-21157355d517" containerID="605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4" exitCode=0
May 26 17:23:48.865723 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.865557    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"
May 26 17:23:48.865723 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.865590    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg" event={"ID":"2a13712f-2bef-4ee2-bc14-21157355d517","Type":"ContainerDied","Data":"605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4"}
May 26 17:23:48.865723 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.865638    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg" event={"ID":"2a13712f-2bef-4ee2-bc14-21157355d517","Type":"ContainerDied","Data":"34f00b1a44cf576476a02706e1673c2e06deaa49e4c18abc62a0545c358cefcd"}
May 26 17:23:48.865723 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.865660    2571 scope.go:117] "RemoveContainer" containerID="605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4"
May 26 17:23:48.876434 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.876347    2571 scope.go:117] "RemoveContainer" containerID="605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4"
May 26 17:23:48.876634 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:23:48.876616    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4\": container with ID starting with 605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4 not found: ID does not exist" containerID="605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4"
May 26 17:23:48.876696 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.876642    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4"} err="failed to get container status \"605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4\": rpc error: code = NotFound desc = could not find container \"605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4\": container with ID starting with 605e7c1f21382070e8d6fd8c6f1857a6e2a61ca213eb5e8356e306da5cb527f4 not found: ID does not exist"
May 26 17:23:48.886720 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.886692    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"]
May 26 17:23:48.890562 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:48.890540    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6dc7cc57cf-l4pwg"]
May 26 17:23:49.838638 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:49.838604    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a13712f-2bef-4ee2-bc14-21157355d517" path="/var/lib/kubelet/pods/2a13712f-2bef-4ee2-bc14-21157355d517/volumes"
May 26 17:23:49.871310 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:49.871278    2571 generic.go:358] "Generic (PLEG): container finished" podID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerID="abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310" exitCode=0
May 26 17:23:49.871474 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:49.871349    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerDied","Data":"abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310"}
May 26 17:23:49.874057 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:49.874035    2571 generic.go:358] "Generic (PLEG): container finished" podID="4516b210-ae0f-4c57-8980-e56081f106f9" containerID="b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d" exitCode=0
May 26 17:23:49.874165 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:49.874095    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerDied","Data":"b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d"}
May 26 17:23:50.765187 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:50.765145    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-mbth5"]
May 26 17:23:50.765382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:50.765348    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5" podUID="463b5edc-b6d9-4f5a-a2b0-80cf86b30575" containerName="postgres" containerID="cri-o://b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3" gracePeriod=30
May 26 17:23:50.769619 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:50.769588    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-r5drb"]
May 26 17:23:50.769814 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:50.769789    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-r5drb" podUID="d211cd58-7973-4a26-ade1-b9b3e5e91416" containerName="redis" containerID="cri-o://bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba" gracePeriod=30
May 26 17:23:51.030689 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.030666    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-r5drb"
May 26 17:23:51.041584 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.041563    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:23:51.062933 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.062900    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqjq7\" (UniqueName: \"kubernetes.io/projected/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-kube-api-access-mqjq7\") pod \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\" (UID: \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\") "
May 26 17:23:51.063154 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.063006    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-postgresdata\") pod \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\" (UID: \"463b5edc-b6d9-4f5a-a2b0-80cf86b30575\") "
May 26 17:23:51.063154 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.063097    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp2dl\" (UniqueName: \"kubernetes.io/projected/d211cd58-7973-4a26-ade1-b9b3e5e91416-kube-api-access-fp2dl\") pod \"d211cd58-7973-4a26-ade1-b9b3e5e91416\" (UID: \"d211cd58-7973-4a26-ade1-b9b3e5e91416\") "
May 26 17:23:51.065377 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.065338    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d211cd58-7973-4a26-ade1-b9b3e5e91416-kube-api-access-fp2dl" (OuterVolumeSpecName: "kube-api-access-fp2dl") pod "d211cd58-7973-4a26-ade1-b9b3e5e91416" (UID: "d211cd58-7973-4a26-ade1-b9b3e5e91416"). InnerVolumeSpecName "kube-api-access-fp2dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:23:51.065485 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.065425    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-kube-api-access-mqjq7" (OuterVolumeSpecName: "kube-api-access-mqjq7") pod "463b5edc-b6d9-4f5a-a2b0-80cf86b30575" (UID: "463b5edc-b6d9-4f5a-a2b0-80cf86b30575"). InnerVolumeSpecName "kube-api-access-mqjq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:23:51.093089 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.093053    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "463b5edc-b6d9-4f5a-a2b0-80cf86b30575" (UID: "463b5edc-b6d9-4f5a-a2b0-80cf86b30575"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:23:51.164385 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.164348    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqjq7\" (UniqueName: \"kubernetes.io/projected/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-kube-api-access-mqjq7\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:23:51.164385 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.164380    2571 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/463b5edc-b6d9-4f5a-a2b0-80cf86b30575-postgresdata\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:23:51.164575 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.164396    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2dl\" (UniqueName: \"kubernetes.io/projected/d211cd58-7973-4a26-ade1-b9b3e5e91416-kube-api-access-fp2dl\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:23:51.882798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.882769    2571 generic.go:358] "Generic (PLEG): container finished" podID="d211cd58-7973-4a26-ade1-b9b3e5e91416" containerID="bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba" exitCode=0
May 26 17:23:51.882970 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.882815    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-r5drb"
May 26 17:23:51.882970 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.882855    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-r5drb" event={"ID":"d211cd58-7973-4a26-ade1-b9b3e5e91416","Type":"ContainerDied","Data":"bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba"}
May 26 17:23:51.882970 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.882900    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-r5drb" event={"ID":"d211cd58-7973-4a26-ade1-b9b3e5e91416","Type":"ContainerDied","Data":"a76c98375747ad9dfba7471a69cab7ed5bf9bbafe1e778b54e48af92252848f2"}
May 26 17:23:51.882970 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.882921    2571 scope.go:117] "RemoveContainer" containerID="bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba"
May 26 17:23:51.884194 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.884172    2571 generic.go:358] "Generic (PLEG): container finished" podID="463b5edc-b6d9-4f5a-a2b0-80cf86b30575" containerID="b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3" exitCode=0
May 26 17:23:51.884274 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.884220    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5"
May 26 17:23:51.884274 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.884262    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5" event={"ID":"463b5edc-b6d9-4f5a-a2b0-80cf86b30575","Type":"ContainerDied","Data":"b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3"}
May 26 17:23:51.884365 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.884297    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-mbth5" event={"ID":"463b5edc-b6d9-4f5a-a2b0-80cf86b30575","Type":"ContainerDied","Data":"cc04bd301382dceb76c33b9b44ada7d4aea346230a17bc2f722fed47bf2124d2"}
May 26 17:23:51.891465 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.891443    2571 scope.go:117] "RemoveContainer" containerID="bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba"
May 26 17:23:51.891718 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:23:51.891694    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba\": container with ID starting with bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba not found: ID does not exist" containerID="bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba"
May 26 17:23:51.891771 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.891723    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba"} err="failed to get container status \"bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba\": rpc error: code = NotFound desc = could not find container \"bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba\": container with ID starting with bb305d24969651399528bfd57a697390d336dc5165493ed297d11182b609eeba not found: ID does not exist"
May 26 17:23:51.891771 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.891742    2571 scope.go:117] "RemoveContainer" containerID="b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3"
May 26 17:23:51.898892 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.898862    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-r5drb"]
May 26 17:23:51.899362 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.899347    2571 scope.go:117] "RemoveContainer" containerID="b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3"
May 26 17:23:51.899604 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:23:51.899582    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3\": container with ID starting with b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3 not found: ID does not exist" containerID="b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3"
May 26 17:23:51.899660 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.899610    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3"} err="failed to get container status \"b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3\": rpc error: code = NotFound desc = could not find container \"b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3\": container with ID starting with b812a096b6dcd2a8ccd5d122ba3aa6845cc75b4ad6ca979e324b8cd5964c09b3 not found: ID does not exist"
May 26 17:23:51.902747 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.902726    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-r5drb"]
May 26 17:23:51.912719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.912681    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-mbth5"]
May 26 17:23:51.914571 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:51.914551    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-mbth5"]
May 26 17:23:53.834235 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:53.834193    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="463b5edc-b6d9-4f5a-a2b0-80cf86b30575" path="/var/lib/kubelet/pods/463b5edc-b6d9-4f5a-a2b0-80cf86b30575/volumes"
May 26 17:23:53.834713 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:53.834682    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d211cd58-7973-4a26-ade1-b9b3e5e91416" path="/var/lib/kubelet/pods/d211cd58-7973-4a26-ade1-b9b3e5e91416/volumes"
May 26 17:23:55.132544 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:55.132502    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused"
May 26 17:23:56.902698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:23:56.902647    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 26 17:24:05.132787 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:05.132741    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused"
May 26 17:24:06.902846 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:06.902800    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 26 17:24:15.132894 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:15.132850    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused"
May 26 17:24:15.133350 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:15.132973    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:24:16.903037 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:16.902967    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 26 17:24:16.903452 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:16.903105    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:24:18.947056 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.947032    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:24:18.950050 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.950031    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:24:18.982155 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.982113    2571 generic.go:358] "Generic (PLEG): container finished" podID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerID="30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed" exitCode=137
May 26 17:24:18.982337 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.982208    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"
May 26 17:24:18.982337 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.982233    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerDied","Data":"30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed"}
May 26 17:24:18.982337 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.982269    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n" event={"ID":"dc138561-06d2-4d1f-8ace-9b33a7146ea3","Type":"ContainerDied","Data":"5efae3795d3f0faac869f2d2f1a7cdcb20bfc70605c4aba5f94243149581f1d3"}
May 26 17:24:18.982337 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.982290    2571 scope.go:117] "RemoveContainer" containerID="abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310"
May 26 17:24:18.984404 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.984380    2571 generic.go:358] "Generic (PLEG): container finished" podID="4516b210-ae0f-4c57-8980-e56081f106f9" containerID="7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6" exitCode=137
May 26 17:24:18.984541 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.984413    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerDied","Data":"7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6"}
May 26 17:24:18.984541 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.984445    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz" event={"ID":"4516b210-ae0f-4c57-8980-e56081f106f9","Type":"ContainerDied","Data":"7ef1f01c0b466bc33912e6a64e4cb668e103286dca6c1cbf62ea5ac0ba17d41e"}
May 26 17:24:18.984541 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.984478    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"
May 26 17:24:18.995005 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:18.994977    2571 scope.go:117] "RemoveContainer" containerID="30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed"
May 26 17:24:19.001928 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.001903    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-online-tls\") pod \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") "
May 26 17:24:19.002094 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.001931    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-registry-tls\") pod \"4516b210-ae0f-4c57-8980-e56081f106f9\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") "
May 26 17:24:19.002094 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.001951    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6rgz\" (UniqueName: \"kubernetes.io/projected/4516b210-ae0f-4c57-8980-e56081f106f9-kube-api-access-w6rgz\") pod \"4516b210-ae0f-4c57-8980-e56081f106f9\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") "
May 26 17:24:19.002094 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.001978    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-registry-tls\") pod \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") "
May 26 17:24:19.002094 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.002079    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4516b210-ae0f-4c57-8980-e56081f106f9-feast-data\") pod \"4516b210-ae0f-4c57-8980-e56081f106f9\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") "
May 26 17:24:19.002352 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.002107    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpxkf\" (UniqueName: \"kubernetes.io/projected/dc138561-06d2-4d1f-8ace-9b33a7146ea3-kube-api-access-xpxkf\") pod \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") "
May 26 17:24:19.002352 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.002131    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-online-tls\") pod \"4516b210-ae0f-4c57-8980-e56081f106f9\" (UID: \"4516b210-ae0f-4c57-8980-e56081f106f9\") "
May 26 17:24:19.002352 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.002173    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc138561-06d2-4d1f-8ace-9b33a7146ea3-feast-data\") pod \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\" (UID: \"dc138561-06d2-4d1f-8ace-9b33a7146ea3\") "
May 26 17:24:19.004232 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.002936    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4516b210-ae0f-4c57-8980-e56081f106f9-feast-data" (OuterVolumeSpecName: "feast-data") pod "4516b210-ae0f-4c57-8980-e56081f106f9" (UID: "4516b210-ae0f-4c57-8980-e56081f106f9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:24:19.004919 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.004809    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dc138561-06d2-4d1f-8ace-9b33a7146ea3" (UID: "dc138561-06d2-4d1f-8ace-9b33a7146ea3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:24:19.004919 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.004889    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4516b210-ae0f-4c57-8980-e56081f106f9" (UID: "4516b210-ae0f-4c57-8980-e56081f106f9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:24:19.005293 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.005211    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc138561-06d2-4d1f-8ace-9b33a7146ea3-kube-api-access-xpxkf" (OuterVolumeSpecName: "kube-api-access-xpxkf") pod "dc138561-06d2-4d1f-8ace-9b33a7146ea3" (UID: "dc138561-06d2-4d1f-8ace-9b33a7146ea3"). InnerVolumeSpecName "kube-api-access-xpxkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:24:19.005293 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.005256    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-online-tls" (OuterVolumeSpecName: "online-tls") pod "4516b210-ae0f-4c57-8980-e56081f106f9" (UID: "4516b210-ae0f-4c57-8980-e56081f106f9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:24:19.005633 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.005608    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-online-tls" (OuterVolumeSpecName: "online-tls") pod "dc138561-06d2-4d1f-8ace-9b33a7146ea3" (UID: "dc138561-06d2-4d1f-8ace-9b33a7146ea3"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:24:19.006483 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.006453    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4516b210-ae0f-4c57-8980-e56081f106f9-kube-api-access-w6rgz" (OuterVolumeSpecName: "kube-api-access-w6rgz") pod "4516b210-ae0f-4c57-8980-e56081f106f9" (UID: "4516b210-ae0f-4c57-8980-e56081f106f9"). InnerVolumeSpecName "kube-api-access-w6rgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:24:19.008131 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.008108    2571 scope.go:117] "RemoveContainer" containerID="e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7"
May 26 17:24:19.015391 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.015352    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc138561-06d2-4d1f-8ace-9b33a7146ea3-feast-data" (OuterVolumeSpecName: "feast-data") pod "dc138561-06d2-4d1f-8ace-9b33a7146ea3" (UID: "dc138561-06d2-4d1f-8ace-9b33a7146ea3"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:24:19.019250 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.019230    2571 scope.go:117] "RemoveContainer" containerID="8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0"
May 26 17:24:19.027684 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.027660    2571 scope.go:117] "RemoveContainer" containerID="abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310"
May 26 17:24:19.028057 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:24:19.028034    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310\": container with ID starting with abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310 not found: ID does not exist" containerID="abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310"
May 26 17:24:19.028143 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.028075    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310"} err="failed to get container status \"abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310\": rpc error: code = NotFound desc = could not find container \"abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310\": container with ID starting with abdef25e7d6be7324762e1e03e5c9bad5fee067404ba457713d1c92346af6310 not found: ID does not exist"
May 26 17:24:19.028143 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.028101    2571 scope.go:117] "RemoveContainer" containerID="30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed"
May 26 17:24:19.028381 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:24:19.028358    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed\": container with ID starting with 30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed not found: ID does not exist" containerID="30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed"
May 26 17:24:19.028489 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.028384    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed"} err="failed to get container status \"30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed\": rpc error: code = NotFound desc = could not find container \"30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed\": container with ID starting with 30a9211b5bc4b6d659e2118f2ba73b7178874f1f2e2ebca2f0412f63fe94faed not found: ID does not exist"
May 26 17:24:19.028489 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.028400    2571 scope.go:117] "RemoveContainer" containerID="e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7"
May 26 17:24:19.028681 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:24:19.028661    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7\": container with ID starting with e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7 not found: ID does not exist" containerID="e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7"
May 26 17:24:19.028723 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.028689    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7"} err="failed to get container status \"e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7\": rpc error: code = NotFound desc = could not find container \"e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7\": container with ID starting with e01dbff5484288cbf2cb394615a98f1ecacfe9166b699e326a9b87b56a14cef7 not found: ID does not exist"
May 26 17:24:19.028723 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.028705    2571 scope.go:117] "RemoveContainer" containerID="8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0"
May 26 17:24:19.028946 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:24:19.028927    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0\": container with ID starting with 8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0 not found: ID does not exist" containerID="8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0"
May 26 17:24:19.029005 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.028949    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0"} err="failed to get container status \"8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0\": rpc error: code = NotFound desc = could not find container \"8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0\": container with ID starting with 8ec11cbd6ff7f38d20b0775145d224736d8bce407b2280832813da91b1dd77e0 not found: ID does not exist"
May 26 17:24:19.029005 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.028962    2571 scope.go:117] "RemoveContainer" containerID="b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d"
May 26 17:24:19.038918 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.038899    2571 scope.go:117] "RemoveContainer" containerID="7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6"
May 26 17:24:19.047339 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.047319    2571 scope.go:117] "RemoveContainer" containerID="300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c"
May 26 17:24:19.055198 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.055178    2571 scope.go:117] "RemoveContainer" containerID="f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858"
May 26 17:24:19.062620 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.062596    2571 scope.go:117] "RemoveContainer" containerID="b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d"
May 26 17:24:19.062924 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:24:19.062904    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d\": container with ID starting with b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d not found: ID does not exist" containerID="b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d"
May 26 17:24:19.062979 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.062932    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d"} err="failed to get container status \"b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d\": rpc error: code = NotFound desc = could not find container \"b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d\": container with ID starting with b2b7c353cb5d2d68b1c20a9fcdb77952dda64f43cc57ae5d669f8121edc1120d not found: ID does not exist"
May 26 17:24:19.062979 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.062952    2571 scope.go:117] "RemoveContainer" containerID="7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6"
May 26 17:24:19.063221 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:24:19.063201    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6\": container with ID starting with 7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6 not found: ID does not exist" containerID="7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6"
May 26 17:24:19.063280 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.063228    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6"} err="failed to get container status \"7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6\": rpc error: code = NotFound desc = could not find container \"7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6\": container with ID starting with 7ced377dda160a305cda033aaf8f31cf712904ee7962a1f37d4cf66bd82504a6 not found: ID does not exist"
May 26 17:24:19.063280 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.063245    2571 scope.go:117] "RemoveContainer" containerID="300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c"
May 26 17:24:19.063495 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:24:19.063476    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c\": container with ID starting with 300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c not found: ID does not exist" containerID="300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c"
May 26 17:24:19.063535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.063519    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c"} err="failed to get container status \"300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c\": rpc error: code = NotFound desc = could not find container \"300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c\": container with ID starting with 300991f37c9b24d88fad479a20a6cb640bcd7c60c580dc877ca52e06acb8cc8c not found: ID does not exist"
May 26 17:24:19.063575 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.063539    2571 scope.go:117] "RemoveContainer" containerID="f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858"
May 26 17:24:19.063758 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:24:19.063736    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858\": container with ID starting with f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858 not found: ID does not exist" containerID="f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858"
May 26 17:24:19.063803 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.063766    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858"} err="failed to get container status \"f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858\": rpc error: code = NotFound desc = could not find container \"f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858\": container with ID starting with f0f45b107312496d904fb17c412122a32fb6a102a11e825411b875da1728f858 not found: ID does not exist"
May 26 17:24:19.103304 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.103266    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4516b210-ae0f-4c57-8980-e56081f106f9-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:24:19.103304 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.103302    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxkf\" (UniqueName: \"kubernetes.io/projected/dc138561-06d2-4d1f-8ace-9b33a7146ea3-kube-api-access-xpxkf\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:24:19.103304 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.103313    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:24:19.103602 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.103322    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc138561-06d2-4d1f-8ace-9b33a7146ea3-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:24:19.103602 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.103331    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:24:19.103602 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.103338    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4516b210-ae0f-4c57-8980-e56081f106f9-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:24:19.103602 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.103347    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rgz\" (UniqueName: \"kubernetes.io/projected/4516b210-ae0f-4c57-8980-e56081f106f9-kube-api-access-w6rgz\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:24:19.103602 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.103355    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc138561-06d2-4d1f-8ace-9b33a7146ea3-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:24:19.308924 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.308892    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"]
May 26 17:24:19.311564 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.311529    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f5794f64d-6rkkz"]
May 26 17:24:19.322895 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.322867    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"]
May 26 17:24:19.327983 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.327957    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5847df4f87-qnl8n"]
May 26 17:24:19.810342 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.810315    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:24:19.813401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.813378    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:24:19.835826 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.835803    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" path="/var/lib/kubelet/pods/4516b210-ae0f-4c57-8980-e56081f106f9/volumes"
May 26 17:24:19.836407 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:19.836386    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" path="/var/lib/kubelet/pods/dc138561-06d2-4d1f-8ace-9b33a7146ea3/volumes"
May 26 17:24:24.160511 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160473    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"]
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160803    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463b5edc-b6d9-4f5a-a2b0-80cf86b30575" containerName="postgres"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160814    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="463b5edc-b6d9-4f5a-a2b0-80cf86b30575" containerName="postgres"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160825    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="feast-apply"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160832    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="feast-apply"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160838    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="online"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160844    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="online"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160852    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="feast-init"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160858    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="feast-init"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160866    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a13712f-2bef-4ee2-bc14-21157355d517" containerName="manager"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160871    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a13712f-2bef-4ee2-bc14-21157355d517" containerName="manager"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160892    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="feast-init"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160897    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="feast-init"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160903    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="feast-apply"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160908    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="feast-apply"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160916    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d211cd58-7973-4a26-ade1-b9b3e5e91416" containerName="redis"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160921    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d211cd58-7973-4a26-ade1-b9b3e5e91416" containerName="redis"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160927    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="online"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160932    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="online"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160941    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="registry"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160946    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="registry"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160953    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="registry"
May 26 17:24:24.160972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.160957    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="registry"
May 26 17:24:24.161705 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.161006    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="registry"
May 26 17:24:24.161705 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.161030    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="d211cd58-7973-4a26-ade1-b9b3e5e91416" containerName="redis"
May 26 17:24:24.161705 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.161039    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a13712f-2bef-4ee2-bc14-21157355d517" containerName="manager"
May 26 17:24:24.161705 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.161046    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="registry"
May 26 17:24:24.161705 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.161059    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="463b5edc-b6d9-4f5a-a2b0-80cf86b30575" containerName="postgres"
May 26 17:24:24.161705 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.161065    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="4516b210-ae0f-4c57-8980-e56081f106f9" containerName="online"
May 26 17:24:24.161705 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.161070    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc138561-06d2-4d1f-8ace-9b33a7146ea3" containerName="online"
May 26 17:24:24.166736 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.166719    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.169400 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.169373    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 26 17:24:24.169581 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.169380    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 26 17:24:24.169581 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.169482    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-cjkk8\""
May 26 17:24:24.169581 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.169525    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 26 17:24:24.169581 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.169556    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 26 17:24:24.169870 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.169856    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 26 17:24:24.169945 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.169929    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 26 17:24:24.173986 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.173963    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"]
May 26 17:24:24.251802 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.251758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.251978 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.251816    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.251978 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.251841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wfg7\" (UniqueName: \"kubernetes.io/projected/c78c9405-531d-49e7-b8d8-902d8033b791-kube-api-access-8wfg7\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.251978 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.251870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c78c9405-531d-49e7-b8d8-902d8033b791-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.251978 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.251898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.252149 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.251977    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.353160 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.353123    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.353355 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.353179    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.353355 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.353212    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.353355 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.353233    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8wfg7\" (UniqueName: \"kubernetes.io/projected/c78c9405-531d-49e7-b8d8-902d8033b791-kube-api-access-8wfg7\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.353355 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.353250    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c78c9405-531d-49e7-b8d8-902d8033b791-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.353588 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.353388    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.353724 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.353693    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c78c9405-531d-49e7-b8d8-902d8033b791-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.355758 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.355728    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.355758 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.355746    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.356002 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.355985    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.356072 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.356004    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.360547 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.360526    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wfg7\" (UniqueName: \"kubernetes.io/projected/c78c9405-531d-49e7-b8d8-902d8033b791-kube-api-access-8wfg7\") pod \"feast-simple-feast-setup-664fcfcc58-mxdgl\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.492057 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.492029    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:24.642243 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:24.642217    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"]
May 26 17:24:24.644866 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:24:24.644836    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc78c9405_531d_49e7_b8d8_902d8033b791.slice/crio-59f87f1a503fb83f8477d7f072b8874519b6eb42c0e68c8011ef584c9da4783c WatchSource:0}: Error finding container 59f87f1a503fb83f8477d7f072b8874519b6eb42c0e68c8011ef584c9da4783c: Status 404 returned error can't find the container with id 59f87f1a503fb83f8477d7f072b8874519b6eb42c0e68c8011ef584c9da4783c
May 26 17:24:25.004368 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:25.004333    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerStarted","Data":"59f87f1a503fb83f8477d7f072b8874519b6eb42c0e68c8011ef584c9da4783c"}
May 26 17:24:40.067124 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:40.067087    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerStarted","Data":"be2b758fc55640fee48a879a07f945ce46c1c5702cf03f388ea856aad5c49e55"}
May 26 17:24:45.089333 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:45.089299    2571 generic.go:358] "Generic (PLEG): container finished" podID="c78c9405-531d-49e7-b8d8-902d8033b791" containerID="be2b758fc55640fee48a879a07f945ce46c1c5702cf03f388ea856aad5c49e55" exitCode=0
May 26 17:24:45.089837 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:45.089382    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerDied","Data":"be2b758fc55640fee48a879a07f945ce46c1c5702cf03f388ea856aad5c49e55"}
May 26 17:24:46.094458 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:46.094418    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerStarted","Data":"5b00ae4c0d23281b64db6f5c8957c59962ad203fd3ae763065b3d1c3a5aeb17d"}
May 26 17:24:51.113423 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:51.113385    2571 generic.go:358] "Generic (PLEG): container finished" podID="c78c9405-531d-49e7-b8d8-902d8033b791" containerID="5b00ae4c0d23281b64db6f5c8957c59962ad203fd3ae763065b3d1c3a5aeb17d" exitCode=0
May 26 17:24:51.113832 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:51.113461    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerDied","Data":"5b00ae4c0d23281b64db6f5c8957c59962ad203fd3ae763065b3d1c3a5aeb17d"}
May 26 17:24:52.121713 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:52.121666    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerStarted","Data":"43d0ee15fe6121e95ef7e1a9c4a5742742d83d7d4c05c241fb3df3678a3ac547"}
May 26 17:24:52.121713 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:52.121716    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerStarted","Data":"66c548a76baf17ddc7d69907c2589c04b77e2bf3cb4489061c470a717afc2d99"}
May 26 17:24:52.122296 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:52.121732    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerStarted","Data":"a4eb90d517556c8fa6471003f798203fb283fc15e6a0de895ef2a7b42d3f0ca5"}
May 26 17:24:52.122296 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:52.121765    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerStarted","Data":"7f0d1eba2ed3674ff428e893918eba2facbadfbce7c6485a1b5239dbb85ea543"}
May 26 17:24:52.146305 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:52.146238    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podStartSLOduration=13.241722726 podStartE2EDuration="28.146216379s" podCreationTimestamp="2026-05-26 17:24:24 +0000 UTC" firstStartedPulling="2026-05-26 17:24:24.646618634 +0000 UTC m=+1205.371708118" lastFinishedPulling="2026-05-26 17:24:39.551112292 +0000 UTC m=+1220.276201771" observedRunningTime="2026-05-26 17:24:52.144111414 +0000 UTC m=+1232.869200939" watchObservedRunningTime="2026-05-26 17:24:52.146216379 +0000 UTC m=+1232.871305883"
May 26 17:24:54.492897 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.492851    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:54.492897 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.492908    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:54.493496 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.492922    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:54.493496 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.492936    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:54.493496 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.492956    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:54.493496 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.492969    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:54.493496 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.492982    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:54.493496 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.492995    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:54.495410 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.495230    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.38:8816: connect: connection refused"
May 26 17:24:54.495410 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.495258    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused"
May 26 17:24:54.495410 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.495267    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.38:6571: connect: connection refused"
May 26 17:24:54.495410 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:54.495357    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused"
May 26 17:24:57.493341 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:57.493311    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:57.493815 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:57.493580    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:57.494029 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:57.493996    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:57.494129 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:57.494110    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:57.494187 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:57.494176    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:57.497872 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:57.497854    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:58.144508 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:58.144481    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:24:58.146776 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:24:58.146750    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:26:00.880225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:00.880186    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"]
May 26 17:26:00.881549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:00.880852    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="online" containerID="cri-o://a4eb90d517556c8fa6471003f798203fb283fc15e6a0de895ef2a7b42d3f0ca5" gracePeriod=30
May 26 17:26:00.881549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:00.881064    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="ui" containerID="cri-o://43d0ee15fe6121e95ef7e1a9c4a5742742d83d7d4c05c241fb3df3678a3ac547" gracePeriod=30
May 26 17:26:00.882160 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:00.881606    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="registry" containerID="cri-o://7f0d1eba2ed3674ff428e893918eba2facbadfbce7c6485a1b5239dbb85ea543" gracePeriod=30
May 26 17:26:00.882160 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:00.881603    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="offline" containerID="cri-o://66c548a76baf17ddc7d69907c2589c04b77e2bf3cb4489061c470a717afc2d99" gracePeriod=30
May 26 17:26:01.263490 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.263452    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"]
May 26 17:26:01.267528 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.267509    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.270300 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.270276    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-5jh7q\""
May 26 17:26:01.275896 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.275871    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"]
May 26 17:26:01.385650 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.385615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39303a5c-75de-48d7-9ce6-0f9d896e6670-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.385852 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.385685    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.385852 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.385711    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.385852 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.385740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8stg\" (UniqueName: \"kubernetes.io/projected/39303a5c-75de-48d7-9ce6-0f9d896e6670-kube-api-access-n8stg\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.385852 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.385763    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.386007 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.385868    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.486802 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.486764    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n8stg\" (UniqueName: \"kubernetes.io/projected/39303a5c-75de-48d7-9ce6-0f9d896e6670-kube-api-access-n8stg\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.486802 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.486809    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.487049 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.486855    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.487049 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:26:01.486967    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 26 17:26:01.487049 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.486997    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39303a5c-75de-48d7-9ce6-0f9d896e6670-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.487179 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:26:01.487062    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-offline-tls podName:39303a5c-75de-48d7-9ce6-0f9d896e6670 nodeName:}" failed. No retries permitted until 2026-05-26 17:26:01.987045979 +0000 UTC m=+1302.712135458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-ls2cp" (UID: "39303a5c-75de-48d7-9ce6-0f9d896e6670") : secret "feast-simple-feast-setup-offline-tls" not found
May 26 17:26:01.487179 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.487130    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.487179 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.487160    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.487396 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:26:01.487233    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:26:01.487396 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:26:01.487283    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-registry-tls podName:39303a5c-75de-48d7-9ce6-0f9d896e6670 nodeName:}" failed. No retries permitted until 2026-05-26 17:26:01.987266253 +0000 UTC m=+1302.712355744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-ls2cp" (UID: "39303a5c-75de-48d7-9ce6-0f9d896e6670") : secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:26:01.487396 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.487360    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39303a5c-75de-48d7-9ce6-0f9d896e6670-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.489638 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.489612    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.489744 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.489612    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.495313 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.495294    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8stg\" (UniqueName: \"kubernetes.io/projected/39303a5c-75de-48d7-9ce6-0f9d896e6670-kube-api-access-n8stg\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.991595 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.991564    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.991964 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.991618    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.994174 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.994148    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:01.994300 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:01.994178    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-ls2cp\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:02.180723 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:02.180679    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:02.310924 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:02.310899    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"]
May 26 17:26:02.312892 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:26:02.312861    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39303a5c_75de_48d7_9ce6_0f9d896e6670.slice/crio-4af4c09b00c40cd719c463fac60306e8127c4ebfaba69560de31a992f41547ed WatchSource:0}: Error finding container 4af4c09b00c40cd719c463fac60306e8127c4ebfaba69560de31a992f41547ed: Status 404 returned error can't find the container with id 4af4c09b00c40cd719c463fac60306e8127c4ebfaba69560de31a992f41547ed
May 26 17:26:02.340968 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:02.340945    2571 generic.go:358] "Generic (PLEG): container finished" podID="c78c9405-531d-49e7-b8d8-902d8033b791" containerID="43d0ee15fe6121e95ef7e1a9c4a5742742d83d7d4c05c241fb3df3678a3ac547" exitCode=0
May 26 17:26:02.340968 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:02.340966    2571 generic.go:358] "Generic (PLEG): container finished" podID="c78c9405-531d-49e7-b8d8-902d8033b791" containerID="a4eb90d517556c8fa6471003f798203fb283fc15e6a0de895ef2a7b42d3f0ca5" exitCode=0
May 26 17:26:02.341199 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:02.341035    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerDied","Data":"43d0ee15fe6121e95ef7e1a9c4a5742742d83d7d4c05c241fb3df3678a3ac547"}
May 26 17:26:02.341199 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:02.341066    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerDied","Data":"a4eb90d517556c8fa6471003f798203fb283fc15e6a0de895ef2a7b42d3f0ca5"}
May 26 17:26:02.342023 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:02.341993    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerStarted","Data":"4af4c09b00c40cd719c463fac60306e8127c4ebfaba69560de31a992f41547ed"}
May 26 17:26:03.349328 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:03.349280    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerStarted","Data":"ce210e01f4d9d81508aeccbcb6c4edb3366926ba0844d0f72756b148673c62fd"}
May 26 17:26:07.363083 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:07.362975    2571 generic.go:358] "Generic (PLEG): container finished" podID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerID="ce210e01f4d9d81508aeccbcb6c4edb3366926ba0844d0f72756b148673c62fd" exitCode=0
May 26 17:26:07.363083 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:07.363042    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerDied","Data":"ce210e01f4d9d81508aeccbcb6c4edb3366926ba0844d0f72756b148673c62fd"}
May 26 17:26:08.144753 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:08.144697    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused"
May 26 17:26:08.144913 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:08.144717    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused"
May 26 17:26:08.368633 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:08.368595    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerStarted","Data":"44749137cb6f9bfed32c2cdbcf5ee8b92b76261540f77d70e041acd3bfd46dd4"}
May 26 17:26:13.384740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:13.384708    2571 generic.go:358] "Generic (PLEG): container finished" podID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerID="44749137cb6f9bfed32c2cdbcf5ee8b92b76261540f77d70e041acd3bfd46dd4" exitCode=0
May 26 17:26:13.385149 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:13.384756    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerDied","Data":"44749137cb6f9bfed32c2cdbcf5ee8b92b76261540f77d70e041acd3bfd46dd4"}
May 26 17:26:14.392718 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:14.392676    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerStarted","Data":"4d6fd9315e32ad8290cde1611f6d2e34929f015bfb551e81889f4a38465d5bf7"}
May 26 17:26:14.393222 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:14.392736    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerStarted","Data":"077557563f49503d592b89bc825f0f36d2eb77c2c3610b2dcf9498285f1113df"}
May 26 17:26:14.393222 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:14.392757    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerStarted","Data":"e8da841ce09bb5232eaac626b74aa0c370976e084d250c583b03d64d02426a2e"}
May 26 17:26:14.393222 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:14.392770    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerStarted","Data":"eb41ed88787e4b78cac7fec80da6fb4759441f890b2f7f1b8fba3082a2d358b6"}
May 26 17:26:14.419348 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:14.419238    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podStartSLOduration=13.419218237 podStartE2EDuration="13.419218237s" podCreationTimestamp="2026-05-26 17:26:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:26:14.416595006 +0000 UTC m=+1315.141684508" watchObservedRunningTime="2026-05-26 17:26:14.419218237 +0000 UTC m=+1315.144307739"
May 26 17:26:17.181227 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:17.181186    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:17.181808 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:17.181772    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:17.181808 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:17.181798    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:17.181808 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:17.181811    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:17.183460 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:17.183427    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused"
May 26 17:26:17.183635 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:17.183455    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.39:6571: connect: connection refused"
May 26 17:26:17.183707 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:17.183473    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.39:8816: connect: connection refused"
May 26 17:26:17.183707 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:17.183549    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused"
May 26 17:26:18.144535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:18.144420    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused"
May 26 17:26:18.144535 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:18.144461    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused"
May 26 17:26:20.182760 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.182725    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.183207 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.182940    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.183207 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.183082    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.183413 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.183392    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.183614 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.183418    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.183614 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.183430    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.183739 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.183692    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.183803 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.183784    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.183914 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.183901    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.187398 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.187381    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.417613 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.417583    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:20.420432 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:20.420413    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:26:28.144885 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:28.144838    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused"
May 26 17:26:28.144885 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:28.144873    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused"
May 26 17:26:28.145434 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:28.145000    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:26:28.145434 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:28.145079    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:26:31.455203 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.455172    2571 generic.go:358] "Generic (PLEG): container finished" podID="c78c9405-531d-49e7-b8d8-902d8033b791" containerID="66c548a76baf17ddc7d69907c2589c04b77e2bf3cb4489061c470a717afc2d99" exitCode=137
May 26 17:26:31.455203 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.455198    2571 generic.go:358] "Generic (PLEG): container finished" podID="c78c9405-531d-49e7-b8d8-902d8033b791" containerID="7f0d1eba2ed3674ff428e893918eba2facbadfbce7c6485a1b5239dbb85ea543" exitCode=137
May 26 17:26:31.455557 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.455237    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerDied","Data":"66c548a76baf17ddc7d69907c2589c04b77e2bf3cb4489061c470a717afc2d99"}
May 26 17:26:31.455557 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.455266    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerDied","Data":"7f0d1eba2ed3674ff428e893918eba2facbadfbce7c6485a1b5239dbb85ea543"}
May 26 17:26:31.537820 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.537797    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:26:31.562880 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.562850    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-ui-tls\") pod \"c78c9405-531d-49e7-b8d8-902d8033b791\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") "
May 26 17:26:31.563072 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.562899    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-registry-tls\") pod \"c78c9405-531d-49e7-b8d8-902d8033b791\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") "
May 26 17:26:31.563072 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.562925    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-offline-tls\") pod \"c78c9405-531d-49e7-b8d8-902d8033b791\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") "
May 26 17:26:31.563072 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.562952    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c78c9405-531d-49e7-b8d8-902d8033b791-feast-data\") pod \"c78c9405-531d-49e7-b8d8-902d8033b791\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") "
May 26 17:26:31.563072 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.562997    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wfg7\" (UniqueName: \"kubernetes.io/projected/c78c9405-531d-49e7-b8d8-902d8033b791-kube-api-access-8wfg7\") pod \"c78c9405-531d-49e7-b8d8-902d8033b791\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") "
May 26 17:26:31.563072 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.563053    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-online-tls\") pod \"c78c9405-531d-49e7-b8d8-902d8033b791\" (UID: \"c78c9405-531d-49e7-b8d8-902d8033b791\") "
May 26 17:26:31.563620 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.563571    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c78c9405-531d-49e7-b8d8-902d8033b791-feast-data" (OuterVolumeSpecName: "feast-data") pod "c78c9405-531d-49e7-b8d8-902d8033b791" (UID: "c78c9405-531d-49e7-b8d8-902d8033b791"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:26:31.565761 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.565721    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78c9405-531d-49e7-b8d8-902d8033b791-kube-api-access-8wfg7" (OuterVolumeSpecName: "kube-api-access-8wfg7") pod "c78c9405-531d-49e7-b8d8-902d8033b791" (UID: "c78c9405-531d-49e7-b8d8-902d8033b791"). InnerVolumeSpecName "kube-api-access-8wfg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:26:31.566157 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.566129    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-online-tls" (OuterVolumeSpecName: "online-tls") pod "c78c9405-531d-49e7-b8d8-902d8033b791" (UID: "c78c9405-531d-49e7-b8d8-902d8033b791"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:26:31.566157 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.566137    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c78c9405-531d-49e7-b8d8-902d8033b791" (UID: "c78c9405-531d-49e7-b8d8-902d8033b791"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:26:31.566307 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.566160    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c78c9405-531d-49e7-b8d8-902d8033b791" (UID: "c78c9405-531d-49e7-b8d8-902d8033b791"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:26:31.566353 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.566330    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c78c9405-531d-49e7-b8d8-902d8033b791" (UID: "c78c9405-531d-49e7-b8d8-902d8033b791"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:26:31.663757 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.663672    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:26:31.663757 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.663701    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:26:31.663757 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.663712    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c78c9405-531d-49e7-b8d8-902d8033b791-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:26:31.663757 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.663722    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wfg7\" (UniqueName: \"kubernetes.io/projected/c78c9405-531d-49e7-b8d8-902d8033b791-kube-api-access-8wfg7\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:26:31.663757 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.663731    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:26:31.663757 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:31.663739    2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c78c9405-531d-49e7-b8d8-902d8033b791-ui-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:26:32.461388 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.461338    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl" event={"ID":"c78c9405-531d-49e7-b8d8-902d8033b791","Type":"ContainerDied","Data":"59f87f1a503fb83f8477d7f072b8874519b6eb42c0e68c8011ef584c9da4783c"}
May 26 17:26:32.461388 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.461376    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"
May 26 17:26:32.461895 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.461401    2571 scope.go:117] "RemoveContainer" containerID="43d0ee15fe6121e95ef7e1a9c4a5742742d83d7d4c05c241fb3df3678a3ac547"
May 26 17:26:32.471095 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.471074    2571 scope.go:117] "RemoveContainer" containerID="66c548a76baf17ddc7d69907c2589c04b77e2bf3cb4489061c470a717afc2d99"
May 26 17:26:32.479215 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.479190    2571 scope.go:117] "RemoveContainer" containerID="a4eb90d517556c8fa6471003f798203fb283fc15e6a0de895ef2a7b42d3f0ca5"
May 26 17:26:32.481251 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.481228    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"]
May 26 17:26:32.485299 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.485274    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-mxdgl"]
May 26 17:26:32.488075 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.488046    2571 scope.go:117] "RemoveContainer" containerID="7f0d1eba2ed3674ff428e893918eba2facbadfbce7c6485a1b5239dbb85ea543"
May 26 17:26:32.496198 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.496172    2571 scope.go:117] "RemoveContainer" containerID="5b00ae4c0d23281b64db6f5c8957c59962ad203fd3ae763065b3d1c3a5aeb17d"
May 26 17:26:32.504154 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:32.504132    2571 scope.go:117] "RemoveContainer" containerID="be2b758fc55640fee48a879a07f945ce46c1c5702cf03f388ea856aad5c49e55"
May 26 17:26:33.834563 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:26:33.834524    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" path="/var/lib/kubelet/pods/c78c9405-531d-49e7-b8d8-902d8033b791/volumes"
May 26 17:27:22.950594 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.950562    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"]
May 26 17:27:22.951007 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.950971    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="online"
May 26 17:27:22.951007 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.950985    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="online"
May 26 17:27:22.951007 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.950998    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="feast-init"
May 26 17:27:22.951007 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951004    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="feast-init"
May 26 17:27:22.951007 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951025    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="registry"
May 26 17:27:22.951007 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951032    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="registry"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951038    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="ui"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951043    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="ui"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951054    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="feast-apply"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951059    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="feast-apply"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951070    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="offline"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951075    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="offline"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951139    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="registry"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951146    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="offline"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951154    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="online"
May 26 17:27:22.951225 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.951161    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="c78c9405-531d-49e7-b8d8-902d8033b791" containerName="ui"
May 26 17:27:22.955655 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.955637    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:22.958529 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.958504    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 26 17:27:22.958737 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.958691    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 26 17:27:22.958737 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.958728    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 26 17:27:22.958944 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.958753    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 26 17:27:22.958944 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.958761    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 26 17:27:22.958944 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.958827    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 26 17:27:22.959143 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.958966    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-f2j29\""
May 26 17:27:22.965574 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:22.965554    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"]
May 26 17:27:23.005575 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.005542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f2dc9886-afc3-44ee-a91a-7b49849209d7-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.005740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.005584    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.005740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.005615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc9886-afc3-44ee-a91a-7b49849209d7-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.005740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.005686    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.005740 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.005722    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.005882 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.005793    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbxwm\" (UniqueName: \"kubernetes.io/projected/f2dc9886-afc3-44ee-a91a-7b49849209d7-kube-api-access-hbxwm\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.106804 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.106769    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.106974 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.106807    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.106974 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.106865    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hbxwm\" (UniqueName: \"kubernetes.io/projected/f2dc9886-afc3-44ee-a91a-7b49849209d7-kube-api-access-hbxwm\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.106974 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.106908    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f2dc9886-afc3-44ee-a91a-7b49849209d7-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.106974 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.106936    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.106974 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.106952    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc9886-afc3-44ee-a91a-7b49849209d7-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.107205 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:27:23.107003    2571 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found
May 26 17:27:23.107205 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:27:23.107093    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-offline-tls podName:f2dc9886-afc3-44ee-a91a-7b49849209d7 nodeName:}" failed. No retries permitted until 2026-05-26 17:27:23.607072506 +0000 UTC m=+1384.332161987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-offline-tls") pod "feast-simple-feast-remote-setup-5cff94746b-4wlrn" (UID: "f2dc9886-afc3-44ee-a91a-7b49849209d7") : secret "feast-simple-feast-remote-setup-offline-tls" not found
May 26 17:27:23.107400 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.107383    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc9886-afc3-44ee-a91a-7b49849209d7-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.107564 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.107543    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f2dc9886-afc3-44ee-a91a-7b49849209d7-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.109338 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.109315    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.109463 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.109343    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.115755 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.115722    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbxwm\" (UniqueName: \"kubernetes.io/projected/f2dc9886-afc3-44ee-a91a-7b49849209d7-kube-api-access-hbxwm\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.612155 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.612107    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.614661 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.614636    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-4wlrn\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:23.865988 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:23.865883    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:24.002720 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:24.002690    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"]
May 26 17:27:24.003406 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:27:24.003384    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2dc9886_afc3_44ee_a91a_7b49849209d7.slice/crio-23fc1040ae5b0446e2f2f598ec7a23cdf56484ee889a129ae21ef9d20f8bb26f WatchSource:0}: Error finding container 23fc1040ae5b0446e2f2f598ec7a23cdf56484ee889a129ae21ef9d20f8bb26f: Status 404 returned error can't find the container with id 23fc1040ae5b0446e2f2f598ec7a23cdf56484ee889a129ae21ef9d20f8bb26f
May 26 17:27:24.005182 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:24.005164    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 17:27:24.628423 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:24.628377    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerStarted","Data":"e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d"}
May 26 17:27:24.628611 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:24.628431    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerStarted","Data":"23fc1040ae5b0446e2f2f598ec7a23cdf56484ee889a129ae21ef9d20f8bb26f"}
May 26 17:27:29.645544 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:29.645506    2571 generic.go:358] "Generic (PLEG): container finished" podID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerID="e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d" exitCode=0
May 26 17:27:29.646041 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:29.645576    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerDied","Data":"e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d"}
May 26 17:27:30.650838 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:30.650803    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerStarted","Data":"e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd"}
May 26 17:27:35.668080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:35.667978    2571 generic.go:358] "Generic (PLEG): container finished" podID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerID="e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd" exitCode=0
May 26 17:27:35.668080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:35.668057    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerDied","Data":"e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd"}
May 26 17:27:36.675561 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:36.675517    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerStarted","Data":"738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0"}
May 26 17:27:36.675955 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:36.675577    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerStarted","Data":"9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30"}
May 26 17:27:36.675955 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:36.675592    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerStarted","Data":"78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d"}
May 26 17:27:36.701453 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:36.701386    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podStartSLOduration=14.701356883999999 podStartE2EDuration="14.701356884s" podCreationTimestamp="2026-05-26 17:27:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:27:36.697791057 +0000 UTC m=+1397.422880559" watchObservedRunningTime="2026-05-26 17:27:36.701356884 +0000 UTC m=+1397.426446386"
May 26 17:27:38.866784 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:38.866694    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:38.866784 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:38.866748    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:38.866784 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:38.866760    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:38.868397 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:38.868358    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused"
May 26 17:27:38.868531 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:38.868393    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.40:8816: connect: connection refused"
May 26 17:27:38.868531 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:38.868403    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused"
May 26 17:27:41.867214 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:41.867185    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:41.867629 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:41.867295    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:41.867791 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:41.867713    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:41.867791 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:41.867757    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:41.867975 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:41.867880    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:41.868130 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:41.868113    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:41.871344 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:41.871322    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:42.697960 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:42.697928    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:27:42.700957 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:27:42.700934    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:28:44.323926 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.323828    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"]
May 26 17:28:44.324717 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.324199    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="online" containerID="cri-o://78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d" gracePeriod=30
May 26 17:28:44.324717 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.324259    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="ui" containerID="cri-o://738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0" gracePeriod=30
May 26 17:28:44.324717 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.324342    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="offline" containerID="cri-o://9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30" gracePeriod=30
May 26 17:28:44.484458 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.484425    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"]
May 26 17:28:44.484791 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.484759    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="registry" containerID="cri-o://eb41ed88787e4b78cac7fec80da6fb4759441f890b2f7f1b8fba3082a2d358b6" gracePeriod=30
May 26 17:28:44.485053 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.484843    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="ui" containerID="cri-o://4d6fd9315e32ad8290cde1611f6d2e34929f015bfb551e81889f4a38465d5bf7" gracePeriod=30
May 26 17:28:44.485162 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.484834    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="online" containerID="cri-o://e8da841ce09bb5232eaac626b74aa0c370976e084d250c583b03d64d02426a2e" gracePeriod=30
May 26 17:28:44.485224 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.484829    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="offline" containerID="cri-o://077557563f49503d592b89bc825f0f36d2eb77c2c3610b2dcf9498285f1113df" gracePeriod=30
May 26 17:28:44.900741 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.900700    2571 generic.go:358] "Generic (PLEG): container finished" podID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerID="738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0" exitCode=0
May 26 17:28:44.900922 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:44.900769    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerDied","Data":"738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0"}
May 26 17:28:45.905517 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:45.905485    2571 generic.go:358] "Generic (PLEG): container finished" podID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerID="78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d" exitCode=0
May 26 17:28:45.905939 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:45.905552    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerDied","Data":"78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d"}
May 26 17:28:45.908080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:45.908060    2571 generic.go:358] "Generic (PLEG): container finished" podID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerID="4d6fd9315e32ad8290cde1611f6d2e34929f015bfb551e81889f4a38465d5bf7" exitCode=0
May 26 17:28:45.908080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:45.908080    2571 generic.go:358] "Generic (PLEG): container finished" podID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerID="e8da841ce09bb5232eaac626b74aa0c370976e084d250c583b03d64d02426a2e" exitCode=0
May 26 17:28:45.908172 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:45.908129    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerDied","Data":"4d6fd9315e32ad8290cde1611f6d2e34929f015bfb551e81889f4a38465d5bf7"}
May 26 17:28:45.908172 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:45.908154    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerDied","Data":"e8da841ce09bb5232eaac626b74aa0c370976e084d250c583b03d64d02426a2e"}
May 26 17:28:50.183637 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:50.183588    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused"
May 26 17:28:50.417908 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:50.417868    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused"
May 26 17:28:51.868428 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:51.868375    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused"
May 26 17:28:52.698934 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:28:52.698876    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused"
May 26 17:29:00.183930 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:00.183882    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused"
May 26 17:29:00.418652 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:00.418605    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused"
May 26 17:29:01.868522 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:01.868471    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused"
May 26 17:29:02.698407 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:02.698367    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused"
May 26 17:29:10.184409 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:10.184361    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused"
May 26 17:29:10.184868 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:10.184490    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:29:10.418159 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:10.418104    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused"
May 26 17:29:10.418365 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:10.418239    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:29:11.868558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:11.868512    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused"
May 26 17:29:11.868994 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:11.868657    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:29:12.698315 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:12.698275    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused"
May 26 17:29:12.698504 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:12.698388    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:29:14.514775 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.514753    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:29:14.570929 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.570900    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f2dc9886-afc3-44ee-a91a-7b49849209d7-registry-tls\") pod \"f2dc9886-afc3-44ee-a91a-7b49849209d7\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") "
May 26 17:29:14.571231 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.570934    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-ui-tls\") pod \"f2dc9886-afc3-44ee-a91a-7b49849209d7\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") "
May 26 17:29:14.571231 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.570960    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbxwm\" (UniqueName: \"kubernetes.io/projected/f2dc9886-afc3-44ee-a91a-7b49849209d7-kube-api-access-hbxwm\") pod \"f2dc9886-afc3-44ee-a91a-7b49849209d7\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") "
May 26 17:29:14.571231 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.571003    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-online-tls\") pod \"f2dc9886-afc3-44ee-a91a-7b49849209d7\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") "
May 26 17:29:14.571231 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.571078    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc9886-afc3-44ee-a91a-7b49849209d7-feast-data\") pod \"f2dc9886-afc3-44ee-a91a-7b49849209d7\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") "
May 26 17:29:14.571231 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.571111    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-offline-tls\") pod \"f2dc9886-afc3-44ee-a91a-7b49849209d7\" (UID: \"f2dc9886-afc3-44ee-a91a-7b49849209d7\") "
May 26 17:29:14.571537 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.571364    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dc9886-afc3-44ee-a91a-7b49849209d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f2dc9886-afc3-44ee-a91a-7b49849209d7" (UID: "f2dc9886-afc3-44ee-a91a-7b49849209d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:29:14.571855 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.571804    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2dc9886-afc3-44ee-a91a-7b49849209d7-feast-data" (OuterVolumeSpecName: "feast-data") pod "f2dc9886-afc3-44ee-a91a-7b49849209d7" (UID: "f2dc9886-afc3-44ee-a91a-7b49849209d7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:29:14.573614 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.573585    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f2dc9886-afc3-44ee-a91a-7b49849209d7" (UID: "f2dc9886-afc3-44ee-a91a-7b49849209d7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:29:14.573614 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.573605    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-online-tls" (OuterVolumeSpecName: "online-tls") pod "f2dc9886-afc3-44ee-a91a-7b49849209d7" (UID: "f2dc9886-afc3-44ee-a91a-7b49849209d7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:29:14.573805 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.573621    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f2dc9886-afc3-44ee-a91a-7b49849209d7" (UID: "f2dc9886-afc3-44ee-a91a-7b49849209d7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:29:14.573805 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.573647    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dc9886-afc3-44ee-a91a-7b49849209d7-kube-api-access-hbxwm" (OuterVolumeSpecName: "kube-api-access-hbxwm") pod "f2dc9886-afc3-44ee-a91a-7b49849209d7" (UID: "f2dc9886-afc3-44ee-a91a-7b49849209d7"). InnerVolumeSpecName "kube-api-access-hbxwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:29:14.671889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.671848    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f2dc9886-afc3-44ee-a91a-7b49849209d7-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:14.671889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.671883    2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-ui-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:14.671889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.671893    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbxwm\" (UniqueName: \"kubernetes.io/projected/f2dc9886-afc3-44ee-a91a-7b49849209d7-kube-api-access-hbxwm\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:14.672175 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.671902    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:14.672175 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.671911    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc9886-afc3-44ee-a91a-7b49849209d7-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:14.672175 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:14.671920    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2dc9886-afc3-44ee-a91a-7b49849209d7-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:15.002118 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.002088    2571 generic.go:358] "Generic (PLEG): container finished" podID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerID="9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30" exitCode=137
May 26 17:29:15.002287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.002169    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"
May 26 17:29:15.002287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.002168    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerDied","Data":"9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30"}
May 26 17:29:15.002287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.002217    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn" event={"ID":"f2dc9886-afc3-44ee-a91a-7b49849209d7","Type":"ContainerDied","Data":"23fc1040ae5b0446e2f2f598ec7a23cdf56484ee889a129ae21ef9d20f8bb26f"}
May 26 17:29:15.002287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.002238    2571 scope.go:117] "RemoveContainer" containerID="738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0"
May 26 17:29:15.005322 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.005288    2571 generic.go:358] "Generic (PLEG): container finished" podID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerID="077557563f49503d592b89bc825f0f36d2eb77c2c3610b2dcf9498285f1113df" exitCode=137
May 26 17:29:15.005322 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.005318    2571 generic.go:358] "Generic (PLEG): container finished" podID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerID="eb41ed88787e4b78cac7fec80da6fb4759441f890b2f7f1b8fba3082a2d358b6" exitCode=137
May 26 17:29:15.005512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.005355    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerDied","Data":"077557563f49503d592b89bc825f0f36d2eb77c2c3610b2dcf9498285f1113df"}
May 26 17:29:15.005512 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.005410    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerDied","Data":"eb41ed88787e4b78cac7fec80da6fb4759441f890b2f7f1b8fba3082a2d358b6"}
May 26 17:29:15.011930 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.011909    2571 scope.go:117] "RemoveContainer" containerID="9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30"
May 26 17:29:15.020793 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.020776    2571 scope.go:117] "RemoveContainer" containerID="78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d"
May 26 17:29:15.025123 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.025079    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"]
May 26 17:29:15.028781 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.028755    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-4wlrn"]
May 26 17:29:15.031750 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.031732    2571 scope.go:117] "RemoveContainer" containerID="e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd"
May 26 17:29:15.040318 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.040291    2571 scope.go:117] "RemoveContainer" containerID="e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d"
May 26 17:29:15.048642 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.048623    2571 scope.go:117] "RemoveContainer" containerID="738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0"
May 26 17:29:15.048973 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:29:15.048955    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0\": container with ID starting with 738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0 not found: ID does not exist" containerID="738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0"
May 26 17:29:15.049080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.048982    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0"} err="failed to get container status \"738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0\": rpc error: code = NotFound desc = could not find container \"738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0\": container with ID starting with 738f5ec534680bae15ea85117b5f373cee380dcac2b94416da04b08a0da0bda0 not found: ID does not exist"
May 26 17:29:15.049080 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.049004    2571 scope.go:117] "RemoveContainer" containerID="9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30"
May 26 17:29:15.049311 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:29:15.049292    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30\": container with ID starting with 9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30 not found: ID does not exist" containerID="9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30"
May 26 17:29:15.049381 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.049320    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30"} err="failed to get container status \"9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30\": rpc error: code = NotFound desc = could not find container \"9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30\": container with ID starting with 9365682dc39d1de7e1caa7dfe707838d4693b7138fd1963a4b3182b7a47d0b30 not found: ID does not exist"
May 26 17:29:15.049381 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.049343    2571 scope.go:117] "RemoveContainer" containerID="78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d"
May 26 17:29:15.049630 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:29:15.049610    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d\": container with ID starting with 78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d not found: ID does not exist" containerID="78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d"
May 26 17:29:15.049685 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.049637    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d"} err="failed to get container status \"78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d\": rpc error: code = NotFound desc = could not find container \"78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d\": container with ID starting with 78146aa202ff7c2e683e25bb48e98207f868e7d5ee05c2174f2196c898faeb4d not found: ID does not exist"
May 26 17:29:15.049685 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.049653    2571 scope.go:117] "RemoveContainer" containerID="e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd"
May 26 17:29:15.049878 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:29:15.049860    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd\": container with ID starting with e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd not found: ID does not exist" containerID="e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd"
May 26 17:29:15.049932 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.049882    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd"} err="failed to get container status \"e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd\": rpc error: code = NotFound desc = could not find container \"e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd\": container with ID starting with e2b4a2a90c1361b3780dd8256a143498a21e4bd5fd660bf423fd4dab2d61f1dd not found: ID does not exist"
May 26 17:29:15.049932 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.049898    2571 scope.go:117] "RemoveContainer" containerID="e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d"
May 26 17:29:15.050155 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:29:15.050110    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d\": container with ID starting with e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d not found: ID does not exist" containerID="e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d"
May 26 17:29:15.050155 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.050145    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d"} err="failed to get container status \"e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d\": rpc error: code = NotFound desc = could not find container \"e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d\": container with ID starting with e3fdde540a347d9a43492a95bd78ed9ac2b1db239de5a5ea4f4119980458cd9d not found: ID does not exist"
May 26 17:29:15.137177 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.137151    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:29:15.175061 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.175028    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8stg\" (UniqueName: \"kubernetes.io/projected/39303a5c-75de-48d7-9ce6-0f9d896e6670-kube-api-access-n8stg\") pod \"39303a5c-75de-48d7-9ce6-0f9d896e6670\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") "
May 26 17:29:15.175253 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.175095    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-ui-tls\") pod \"39303a5c-75de-48d7-9ce6-0f9d896e6670\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") "
May 26 17:29:15.175253 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.175122    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-online-tls\") pod \"39303a5c-75de-48d7-9ce6-0f9d896e6670\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") "
May 26 17:29:15.175253 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.175147    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39303a5c-75de-48d7-9ce6-0f9d896e6670-feast-data\") pod \"39303a5c-75de-48d7-9ce6-0f9d896e6670\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") "
May 26 17:29:15.175253 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.175168    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-registry-tls\") pod \"39303a5c-75de-48d7-9ce6-0f9d896e6670\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") "
May 26 17:29:15.175253 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.175248    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-offline-tls\") pod \"39303a5c-75de-48d7-9ce6-0f9d896e6670\" (UID: \"39303a5c-75de-48d7-9ce6-0f9d896e6670\") "
May 26 17:29:15.175728 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.175694    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39303a5c-75de-48d7-9ce6-0f9d896e6670-feast-data" (OuterVolumeSpecName: "feast-data") pod "39303a5c-75de-48d7-9ce6-0f9d896e6670" (UID: "39303a5c-75de-48d7-9ce6-0f9d896e6670"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:29:15.177350 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.177317    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39303a5c-75de-48d7-9ce6-0f9d896e6670-kube-api-access-n8stg" (OuterVolumeSpecName: "kube-api-access-n8stg") pod "39303a5c-75de-48d7-9ce6-0f9d896e6670" (UID: "39303a5c-75de-48d7-9ce6-0f9d896e6670"). InnerVolumeSpecName "kube-api-access-n8stg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:29:15.177480 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.177382    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "39303a5c-75de-48d7-9ce6-0f9d896e6670" (UID: "39303a5c-75de-48d7-9ce6-0f9d896e6670"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:29:15.177569 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.177554    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "39303a5c-75de-48d7-9ce6-0f9d896e6670" (UID: "39303a5c-75de-48d7-9ce6-0f9d896e6670"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:29:15.177665 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.177650    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-online-tls" (OuterVolumeSpecName: "online-tls") pod "39303a5c-75de-48d7-9ce6-0f9d896e6670" (UID: "39303a5c-75de-48d7-9ce6-0f9d896e6670"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:29:15.177804 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.177789    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "39303a5c-75de-48d7-9ce6-0f9d896e6670" (UID: "39303a5c-75de-48d7-9ce6-0f9d896e6670"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:29:15.276172 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.276085    2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-ui-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:15.276172 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.276116    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:15.276172 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.276125    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39303a5c-75de-48d7-9ce6-0f9d896e6670-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:15.276172 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.276133    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:15.276172 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.276142    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39303a5c-75de-48d7-9ce6-0f9d896e6670-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:15.276172 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.276151    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8stg\" (UniqueName: \"kubernetes.io/projected/39303a5c-75de-48d7-9ce6-0f9d896e6670-kube-api-access-n8stg\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:29:15.835098 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:15.835063    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" path="/var/lib/kubelet/pods/f2dc9886-afc3-44ee-a91a-7b49849209d7/volumes"
May 26 17:29:16.011599 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.011569    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp" event={"ID":"39303a5c-75de-48d7-9ce6-0f9d896e6670","Type":"ContainerDied","Data":"4af4c09b00c40cd719c463fac60306e8127c4ebfaba69560de31a992f41547ed"}
May 26 17:29:16.011759 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.011578    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"
May 26 17:29:16.011759 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.011613    2571 scope.go:117] "RemoveContainer" containerID="4d6fd9315e32ad8290cde1611f6d2e34929f015bfb551e81889f4a38465d5bf7"
May 26 17:29:16.019719 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.019699    2571 scope.go:117] "RemoveContainer" containerID="077557563f49503d592b89bc825f0f36d2eb77c2c3610b2dcf9498285f1113df"
May 26 17:29:16.026983 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.026963    2571 scope.go:117] "RemoveContainer" containerID="e8da841ce09bb5232eaac626b74aa0c370976e084d250c583b03d64d02426a2e"
May 26 17:29:16.032475 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.032449    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"]
May 26 17:29:16.034940 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.034901    2571 scope.go:117] "RemoveContainer" containerID="eb41ed88787e4b78cac7fec80da6fb4759441f890b2f7f1b8fba3082a2d358b6"
May 26 17:29:16.037289 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.037266    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-ls2cp"]
May 26 17:29:16.042473 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.042458    2571 scope.go:117] "RemoveContainer" containerID="44749137cb6f9bfed32c2cdbcf5ee8b92b76261540f77d70e041acd3bfd46dd4"
May 26 17:29:16.049475 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:16.049458    2571 scope.go:117] "RemoveContainer" containerID="ce210e01f4d9d81508aeccbcb6c4edb3366926ba0844d0f72756b148673c62fd"
May 26 17:29:17.835282 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:17.835243    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" path="/var/lib/kubelet/pods/39303a5c-75de-48d7-9ce6-0f9d896e6670/volumes"
May 26 17:29:19.834506 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:19.834478    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:29:19.838046 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:19.838004    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:29:55.080966 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.080931    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"]
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081282    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="online"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081293    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="online"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081303    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="online"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081308    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="online"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081319    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="ui"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081325    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="ui"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081333    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="feast-apply"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081339    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="feast-apply"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081346    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="offline"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081351    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="offline"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081361    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="offline"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081366    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="offline"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081381    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="registry"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081386    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="registry"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081392    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="feast-apply"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081397    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="feast-apply"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081403    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="feast-init"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081407    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="feast-init"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081414    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="ui"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081419    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="ui"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081425    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="feast-init"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081430    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="feast-init"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081481    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="offline"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081489    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="online"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081495    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="ui"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081501    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="ui"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081510    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="online"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081516    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2dc9886-afc3-44ee-a91a-7b49849209d7" containerName="offline"
May 26 17:29:55.083546 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.081525    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="39303a5c-75de-48d7-9ce6-0f9d896e6670" containerName="registry"
May 26 17:29:55.084909 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.084892    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.088313 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.088290    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 26 17:29:55.088466 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.088444    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-4fc2f\""
May 26 17:29:55.088521 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.088473    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 26 17:29:55.088733 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.088718    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 26 17:29:55.088803 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.088762    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 26 17:29:55.088903 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.088890    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 26 17:29:55.088970 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.088953    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 26 17:29:55.107000 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.106968    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"]
May 26 17:29:55.223747 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.223715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e064333b-d50e-49b8-80a3-67977f3ae1a7-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.223747 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.223754    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.224053 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.223781    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.224053 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.223894    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.224053 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.223950    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.224053 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.223985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtnl\" (UniqueName: \"kubernetes.io/projected/e064333b-d50e-49b8-80a3-67977f3ae1a7-kube-api-access-9mtnl\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.324651 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.324611    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e064333b-d50e-49b8-80a3-67977f3ae1a7-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.324651 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.324655    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.324883 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.324682    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.324883 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.324722    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.324883 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.324765    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.324883 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.324796    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtnl\" (UniqueName: \"kubernetes.io/projected/e064333b-d50e-49b8-80a3-67977f3ae1a7-kube-api-access-9mtnl\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.325075 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:29:55.324919    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 26 17:29:55.325075 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:29:55.325000    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-online-tls podName:e064333b-d50e-49b8-80a3-67977f3ae1a7 nodeName:}" failed. No retries permitted until 2026-05-26 17:29:55.824977559 +0000 UTC m=+1536.550067056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-online-tls") pod "feast-simple-feast-setup-664fcfcc58-fswpw" (UID: "e064333b-d50e-49b8-80a3-67977f3ae1a7") : secret "feast-simple-feast-setup-online-tls" not found
May 26 17:29:55.325188 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.325129    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e064333b-d50e-49b8-80a3-67977f3ae1a7-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.327273 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.327243    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.327411 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.327304    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.327411 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.327359    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.333038 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.332964    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtnl\" (UniqueName: \"kubernetes.io/projected/e064333b-d50e-49b8-80a3-67977f3ae1a7-kube-api-access-9mtnl\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.829578 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.829537    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:55.832081 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:55.832053    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fswpw\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:56.007436 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:56.007376    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:29:56.145972 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:56.145946    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"]
May 26 17:29:56.146878 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:29:56.146853    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode064333b_d50e_49b8_80a3_67977f3ae1a7.slice/crio-7afb87a46ecaa18dc5b80d49f6a0646c5552c7f104dd32d575eedcdc56e4f74f WatchSource:0}: Error finding container 7afb87a46ecaa18dc5b80d49f6a0646c5552c7f104dd32d575eedcdc56e4f74f: Status 404 returned error can't find the container with id 7afb87a46ecaa18dc5b80d49f6a0646c5552c7f104dd32d575eedcdc56e4f74f
May 26 17:29:57.147228 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:57.147182    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerStarted","Data":"19f3251c1960e27bcf5643db19fcf00fac64acec89a67af8f2bf027934a85a80"}
May 26 17:29:57.147228 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:29:57.147231    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerStarted","Data":"7afb87a46ecaa18dc5b80d49f6a0646c5552c7f104dd32d575eedcdc56e4f74f"}
May 26 17:30:01.160549 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:01.160514    2571 generic.go:358] "Generic (PLEG): container finished" podID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerID="19f3251c1960e27bcf5643db19fcf00fac64acec89a67af8f2bf027934a85a80" exitCode=0
May 26 17:30:01.160940 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:01.160573    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerDied","Data":"19f3251c1960e27bcf5643db19fcf00fac64acec89a67af8f2bf027934a85a80"}
May 26 17:30:02.165622 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:02.165579    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerStarted","Data":"95498eb381d42d6be9358e8b38a8b24add8a474939415409fc6151cc8d8ceedc"}
May 26 17:30:07.182814 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:07.182723    2571 generic.go:358] "Generic (PLEG): container finished" podID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerID="95498eb381d42d6be9358e8b38a8b24add8a474939415409fc6151cc8d8ceedc" exitCode=0
May 26 17:30:07.182814 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:07.182796    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerDied","Data":"95498eb381d42d6be9358e8b38a8b24add8a474939415409fc6151cc8d8ceedc"}
May 26 17:30:08.190731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:08.190692    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerStarted","Data":"8c8599dd416fd150baeed079220c99442a95f2ef2ef3ad0da004fa29edcc8fe7"}
May 26 17:30:08.191248 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:08.190745    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerStarted","Data":"36c05fa344f2d72cf9311e15c59eb488ec0751a70a4e8869fb05397c83c3fc22"}
May 26 17:30:08.191248 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:08.190762    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerStarted","Data":"420a0cea3b85beb0e32b3e98edfa9a20309dc854b02811f36ed700d20e7bd5e6"}
May 26 17:30:08.191248 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:08.190774    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerStarted","Data":"28c4d7be8a3b8fb51ef49e35059220872ef5f007cbf4785e7a60127251e465f6"}
May 26 17:30:08.217614 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:08.217551    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podStartSLOduration=13.21752821 podStartE2EDuration="13.21752821s" podCreationTimestamp="2026-05-26 17:29:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:30:08.215625506 +0000 UTC m=+1548.940715020" watchObservedRunningTime="2026-05-26 17:30:08.21752821 +0000 UTC m=+1548.942617715"
May 26 17:30:11.008480 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:11.008436    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:11.008480 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:11.008487    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:11.009059 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:11.008502    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:11.009059 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:11.008515    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:11.010666 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:11.010614    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.41:6571: connect: connection refused"
May 26 17:30:11.010666 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:11.010644    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.41:8443: connect: connection refused"
May 26 17:30:11.010885 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:11.010706    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.41:6567/health\": dial tcp 10.134.0.41:6567: connect: connection refused"
May 26 17:30:11.010885 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:11.010652    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.41:8816: connect: connection refused"
May 26 17:30:14.008924 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.008884    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.009441 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.008992    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.009441 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.009230    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.009574 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.009546    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.009626 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.009574    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.009626 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.009587    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.009799 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.009750    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.009866 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.009808    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.009944 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.009927    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.013551 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.013533    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.213240 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.213210    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:30:14.216110 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:30:14.216085    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:31:15.261901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.261802    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"]
May 26 17:31:15.262483 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.262194    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="registry" containerID="cri-o://28c4d7be8a3b8fb51ef49e35059220872ef5f007cbf4785e7a60127251e465f6" gracePeriod=30
May 26 17:31:15.262483 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.262254    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="online" containerID="cri-o://420a0cea3b85beb0e32b3e98edfa9a20309dc854b02811f36ed700d20e7bd5e6" gracePeriod=30
May 26 17:31:15.262483 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.262253    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="ui" containerID="cri-o://8c8599dd416fd150baeed079220c99442a95f2ef2ef3ad0da004fa29edcc8fe7" gracePeriod=30
May 26 17:31:15.262483 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.262223    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="offline" containerID="cri-o://36c05fa344f2d72cf9311e15c59eb488ec0751a70a4e8869fb05397c83c3fc22" gracePeriod=30
May 26 17:31:15.647638 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.647532    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"]
May 26 17:31:15.651870 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.651841    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.654727 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.654699    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-jqtn8\""
May 26 17:31:15.663693 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.663662    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"]
May 26 17:31:15.734180 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.734142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.734180 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.734185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.734425 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.734208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6cv9\" (UniqueName: \"kubernetes.io/projected/95e1ea7e-80d5-493a-b114-e452525d55a7-kube-api-access-g6cv9\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.734425 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.734383    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.734516 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.734439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95e1ea7e-80d5-493a-b114-e452525d55a7-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.734516 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.734480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.835289 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.835256    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.835469 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.835302    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.835469 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:31:15.835391    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:31:15.835469 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:31:15.835448    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls podName:95e1ea7e-80d5-493a-b114-e452525d55a7 nodeName:}" failed. No retries permitted until 2026-05-26 17:31:16.335427705 +0000 UTC m=+1617.060517188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-gg4kq" (UID: "95e1ea7e-80d5-493a-b114-e452525d55a7") : secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:31:15.835469 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.835463    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6cv9\" (UniqueName: \"kubernetes.io/projected/95e1ea7e-80d5-493a-b114-e452525d55a7-kube-api-access-g6cv9\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.835666 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.835538    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.835666 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.835569    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95e1ea7e-80d5-493a-b114-e452525d55a7-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.835666 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.835603    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.835818 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:31:15.835683    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 26 17:31:15.835818 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:31:15.835762    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-ui-tls podName:95e1ea7e-80d5-493a-b114-e452525d55a7 nodeName:}" failed. No retries permitted until 2026-05-26 17:31:16.335746129 +0000 UTC m=+1617.060835623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-gg4kq" (UID: "95e1ea7e-80d5-493a-b114-e452525d55a7") : secret "feast-simple-feast-setup-ui-tls" not found
May 26 17:31:15.835993 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.835962    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95e1ea7e-80d5-493a-b114-e452525d55a7-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.837798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.837772    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.838037 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.838000    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:15.846328 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:15.846305    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6cv9\" (UniqueName: \"kubernetes.io/projected/95e1ea7e-80d5-493a-b114-e452525d55a7-kube-api-access-g6cv9\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:16.341085 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:16.341043    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:16.341490 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:16.341119    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:16.341490 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:31:16.341247    2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:31:16.341490 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:31:16.341314    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls podName:95e1ea7e-80d5-493a-b114-e452525d55a7 nodeName:}" failed. No retries permitted until 2026-05-26 17:31:17.341298799 +0000 UTC m=+1618.066388278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-gg4kq" (UID: "95e1ea7e-80d5-493a-b114-e452525d55a7") : secret "feast-simple-feast-setup-registry-tls" not found
May 26 17:31:16.343525 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:16.343500    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:16.413473 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:16.413439    2571 generic.go:358] "Generic (PLEG): container finished" podID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerID="8c8599dd416fd150baeed079220c99442a95f2ef2ef3ad0da004fa29edcc8fe7" exitCode=0
May 26 17:31:16.413473 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:16.413468    2571 generic.go:358] "Generic (PLEG): container finished" podID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerID="420a0cea3b85beb0e32b3e98edfa9a20309dc854b02811f36ed700d20e7bd5e6" exitCode=0
May 26 17:31:16.413672 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:16.413519    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerDied","Data":"8c8599dd416fd150baeed079220c99442a95f2ef2ef3ad0da004fa29edcc8fe7"}
May 26 17:31:16.413672 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:16.413552    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerDied","Data":"420a0cea3b85beb0e32b3e98edfa9a20309dc854b02811f36ed700d20e7bd5e6"}
May 26 17:31:17.350777 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:17.350738    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:17.353145 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:17.353120    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gg4kq\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:17.464300 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:17.464257    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:17.597877 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:17.597847    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"]
May 26 17:31:17.598588 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:31:17.598561    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95e1ea7e_80d5_493a_b114_e452525d55a7.slice/crio-f2c52c0f3ba50b13ea95d184d87da54560d946cdec75217e18d19af7d8c28aed WatchSource:0}: Error finding container f2c52c0f3ba50b13ea95d184d87da54560d946cdec75217e18d19af7d8c28aed: Status 404 returned error can't find the container with id f2c52c0f3ba50b13ea95d184d87da54560d946cdec75217e18d19af7d8c28aed
May 26 17:31:18.421899 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:18.421858    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerStarted","Data":"1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2"}
May 26 17:31:18.421899 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:18.421898    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerStarted","Data":"f2c52c0f3ba50b13ea95d184d87da54560d946cdec75217e18d19af7d8c28aed"}
May 26 17:31:22.435982 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:22.435946    2571 generic.go:358] "Generic (PLEG): container finished" podID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerID="1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2" exitCode=0
May 26 17:31:22.436378 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:22.436040    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerDied","Data":"1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2"}
May 26 17:31:23.440959 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:23.440924    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerStarted","Data":"a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70"}
May 26 17:31:24.009841 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:24.009788    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.41:8443: connect: connection refused"
May 26 17:31:24.214146 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:24.214093    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.41:6567/health\": dial tcp 10.134.0.41:6567: connect: connection refused"
May 26 17:31:28.459838 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:28.459801    2571 generic.go:358] "Generic (PLEG): container finished" podID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerID="a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70" exitCode=0
May 26 17:31:28.460224 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:28.459874    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerDied","Data":"a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70"}
May 26 17:31:29.469226 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:29.469180    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerStarted","Data":"88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49"}
May 26 17:31:29.469226 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:29.469235    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerStarted","Data":"67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e"}
May 26 17:31:29.469815 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:29.469252    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerStarted","Data":"6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f"}
May 26 17:31:29.469815 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:29.469266    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerStarted","Data":"c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039"}
May 26 17:31:29.493693 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:29.493631    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podStartSLOduration=14.493610477 podStartE2EDuration="14.493610477s" podCreationTimestamp="2026-05-26 17:31:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:31:29.493349971 +0000 UTC m=+1630.218439473" watchObservedRunningTime="2026-05-26 17:31:29.493610477 +0000 UTC m=+1630.218699979"
May 26 17:31:32.464747 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:32.464667    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:32.464747 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:32.464715    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:32.464747 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:32.464728    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:32.464747 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:32.464740    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:32.467119 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:32.466987    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.42:8443: connect: connection refused"
May 26 17:31:32.467119 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:32.467043    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.42:6567/health\": dial tcp 10.134.0.42:6567: connect: connection refused"
May 26 17:31:32.467119 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:32.467070    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.42:6571: connect: connection refused"
May 26 17:31:32.467119 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:32.467006    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.42:8816: connect: connection refused"
May 26 17:31:34.009787 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:34.009732    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.41:8443: connect: connection refused"
May 26 17:31:34.214312 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:34.214266    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.41:6567/health\": dial tcp 10.134.0.41:6567: connect: connection refused"
May 26 17:31:35.466145 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.466111    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.466547 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.466366    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.466547 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.466460    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.466825 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.466799    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.466946 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.466834    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.466946 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.466847    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.467121 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.467081    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.467210 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.467193    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.467287 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.467272    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.470760 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.470747    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.492664 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.492627    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:35.495572 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:35.495540    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:31:44.010094 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:44.010047    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.41:8443: connect: connection refused"
May 26 17:31:44.010594 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:44.010185    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:31:44.213599 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:44.213556    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.41:6567/health\": dial tcp 10.134.0.41:6567: connect: connection refused"
May 26 17:31:44.213780 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:44.213678    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:31:45.527481 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:45.527392    2571 generic.go:358] "Generic (PLEG): container finished" podID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerID="36c05fa344f2d72cf9311e15c59eb488ec0751a70a4e8869fb05397c83c3fc22" exitCode=137
May 26 17:31:45.527481 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:45.527418    2571 generic.go:358] "Generic (PLEG): container finished" podID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerID="28c4d7be8a3b8fb51ef49e35059220872ef5f007cbf4785e7a60127251e465f6" exitCode=137
May 26 17:31:45.527481 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:45.527464    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerDied","Data":"36c05fa344f2d72cf9311e15c59eb488ec0751a70a4e8869fb05397c83c3fc22"}
May 26 17:31:45.527885 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:45.527504    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerDied","Data":"28c4d7be8a3b8fb51ef49e35059220872ef5f007cbf4785e7a60127251e465f6"}
May 26 17:31:45.914693 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:45.914666    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:31:46.108845 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.108742    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mtnl\" (UniqueName: \"kubernetes.io/projected/e064333b-d50e-49b8-80a3-67977f3ae1a7-kube-api-access-9mtnl\") pod \"e064333b-d50e-49b8-80a3-67977f3ae1a7\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") "
May 26 17:31:46.108845 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.108793    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-online-tls\") pod \"e064333b-d50e-49b8-80a3-67977f3ae1a7\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") "
May 26 17:31:46.108845 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.108813    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-offline-tls\") pod \"e064333b-d50e-49b8-80a3-67977f3ae1a7\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") "
May 26 17:31:46.109273 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.108874    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e064333b-d50e-49b8-80a3-67977f3ae1a7-feast-data\") pod \"e064333b-d50e-49b8-80a3-67977f3ae1a7\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") "
May 26 17:31:46.109273 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.108899    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-registry-tls\") pod \"e064333b-d50e-49b8-80a3-67977f3ae1a7\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") "
May 26 17:31:46.109273 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.108954    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-ui-tls\") pod \"e064333b-d50e-49b8-80a3-67977f3ae1a7\" (UID: \"e064333b-d50e-49b8-80a3-67977f3ae1a7\") "
May 26 17:31:46.109508 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.109481    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e064333b-d50e-49b8-80a3-67977f3ae1a7-feast-data" (OuterVolumeSpecName: "feast-data") pod "e064333b-d50e-49b8-80a3-67977f3ae1a7" (UID: "e064333b-d50e-49b8-80a3-67977f3ae1a7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:31:46.111221 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.111181    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-online-tls" (OuterVolumeSpecName: "online-tls") pod "e064333b-d50e-49b8-80a3-67977f3ae1a7" (UID: "e064333b-d50e-49b8-80a3-67977f3ae1a7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:31:46.111317 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.111236    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e064333b-d50e-49b8-80a3-67977f3ae1a7-kube-api-access-9mtnl" (OuterVolumeSpecName: "kube-api-access-9mtnl") pod "e064333b-d50e-49b8-80a3-67977f3ae1a7" (UID: "e064333b-d50e-49b8-80a3-67977f3ae1a7"). InnerVolumeSpecName "kube-api-access-9mtnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:31:46.111317 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.111288    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e064333b-d50e-49b8-80a3-67977f3ae1a7" (UID: "e064333b-d50e-49b8-80a3-67977f3ae1a7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:31:46.111418 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.111399    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e064333b-d50e-49b8-80a3-67977f3ae1a7" (UID: "e064333b-d50e-49b8-80a3-67977f3ae1a7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:31:46.111463 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.111412    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e064333b-d50e-49b8-80a3-67977f3ae1a7" (UID: "e064333b-d50e-49b8-80a3-67977f3ae1a7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:31:46.210563 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.210524    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtnl\" (UniqueName: \"kubernetes.io/projected/e064333b-d50e-49b8-80a3-67977f3ae1a7-kube-api-access-9mtnl\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:31:46.210563 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.210567    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:31:46.210806 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.210583    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:31:46.210806 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.210597    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e064333b-d50e-49b8-80a3-67977f3ae1a7-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:31:46.210806 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.210611    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:31:46.210806 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.210623    2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e064333b-d50e-49b8-80a3-67977f3ae1a7-ui-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:31:46.533025 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.532996    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"
May 26 17:31:46.533524 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.532992    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw" event={"ID":"e064333b-d50e-49b8-80a3-67977f3ae1a7","Type":"ContainerDied","Data":"7afb87a46ecaa18dc5b80d49f6a0646c5552c7f104dd32d575eedcdc56e4f74f"}
May 26 17:31:46.533524 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.533140    2571 scope.go:117] "RemoveContainer" containerID="8c8599dd416fd150baeed079220c99442a95f2ef2ef3ad0da004fa29edcc8fe7"
May 26 17:31:46.542187 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.542170    2571 scope.go:117] "RemoveContainer" containerID="36c05fa344f2d72cf9311e15c59eb488ec0751a70a4e8869fb05397c83c3fc22"
May 26 17:31:46.549675 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.549655    2571 scope.go:117] "RemoveContainer" containerID="420a0cea3b85beb0e32b3e98edfa9a20309dc854b02811f36ed700d20e7bd5e6"
May 26 17:31:46.555770 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.555741    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"]
May 26 17:31:46.558159 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.558142    2571 scope.go:117] "RemoveContainer" containerID="28c4d7be8a3b8fb51ef49e35059220872ef5f007cbf4785e7a60127251e465f6"
May 26 17:31:46.559401 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.559379    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fswpw"]
May 26 17:31:46.565708 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.565688    2571 scope.go:117] "RemoveContainer" containerID="95498eb381d42d6be9358e8b38a8b24add8a474939415409fc6151cc8d8ceedc"
May 26 17:31:46.572817 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:46.572802    2571 scope.go:117] "RemoveContainer" containerID="19f3251c1960e27bcf5643db19fcf00fac64acec89a67af8f2bf027934a85a80"
May 26 17:31:47.834921 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:31:47.834891    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" path="/var/lib/kubelet/pods/e064333b-d50e-49b8-80a3-67977f3ae1a7/volumes"
May 26 17:32:37.150345 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150308    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"]
May 26 17:32:37.150798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150747    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="feast-apply"
May 26 17:32:37.150798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150760    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="feast-apply"
May 26 17:32:37.150798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150775    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="offline"
May 26 17:32:37.150798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150781    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="offline"
May 26 17:32:37.150798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150795    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="online"
May 26 17:32:37.150798 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150801    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="online"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150808    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="feast-init"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150814    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="feast-init"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150821    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="registry"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150826    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="registry"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150831    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="ui"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150836    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="ui"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150892    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="registry"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150901    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="offline"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150908    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="ui"
May 26 17:32:37.151030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.150915    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="e064333b-d50e-49b8-80a3-67977f3ae1a7" containerName="online"
May 26 17:32:37.155600 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.155581    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.158464 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.158442    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 26 17:32:37.158627 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.158547    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 26 17:32:37.158627 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.158557    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 26 17:32:37.158627 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.158600    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 26 17:32:37.158954 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.158940    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 26 17:32:37.159043 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.158945    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 26 17:32:37.159648 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.159624    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-7r6j9\""
May 26 17:32:37.166238 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.166217    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"]
May 26 17:32:37.230351 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.230314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.230351 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.230352    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.230571 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.230388    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/248b4148-0459-48bc-a321-917204ebc623-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.230571 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.230465    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcwgl\" (UniqueName: \"kubernetes.io/projected/248b4148-0459-48bc-a321-917204ebc623-kube-api-access-kcwgl\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.230571 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.230532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/248b4148-0459-48bc-a321-917204ebc623-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.230676 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.230571    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.331845 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.331794    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/248b4148-0459-48bc-a321-917204ebc623-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.332083 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.331853    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kcwgl\" (UniqueName: \"kubernetes.io/projected/248b4148-0459-48bc-a321-917204ebc623-kube-api-access-kcwgl\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.332083 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.331939    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/248b4148-0459-48bc-a321-917204ebc623-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.332083 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.332043    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.332280 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.332088    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.332280 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.332115    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.332379 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.332300    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/248b4148-0459-48bc-a321-917204ebc623-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.332758 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.332729    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/248b4148-0459-48bc-a321-917204ebc623-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.334766 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.334739    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.334877 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.334743    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.334877 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.334776    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.339845 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.339820    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcwgl\" (UniqueName: \"kubernetes.io/projected/248b4148-0459-48bc-a321-917204ebc623-kube-api-access-kcwgl\") pod \"feast-simple-feast-remote-setup-5cff94746b-j62cj\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.465500 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.465398    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:37.607263 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.607233    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"]
May 26 17:32:37.608581 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:32:37.608542    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod248b4148_0459_48bc_a321_917204ebc623.slice/crio-8238a7f1d6a7a2e7cb027a9d5c8e23ff36771df8830ce0271c1a54fda3ef99e0 WatchSource:0}: Error finding container 8238a7f1d6a7a2e7cb027a9d5c8e23ff36771df8830ce0271c1a54fda3ef99e0: Status 404 returned error can't find the container with id 8238a7f1d6a7a2e7cb027a9d5c8e23ff36771df8830ce0271c1a54fda3ef99e0
May 26 17:32:37.610334 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.610317    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 17:32:37.697894 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.697857    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerStarted","Data":"ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1"}
May 26 17:32:37.697894 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:37.697898    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerStarted","Data":"8238a7f1d6a7a2e7cb027a9d5c8e23ff36771df8830ce0271c1a54fda3ef99e0"}
May 26 17:32:42.714772 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:42.714677    2571 generic.go:358] "Generic (PLEG): container finished" podID="248b4148-0459-48bc-a321-917204ebc623" containerID="ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1" exitCode=0
May 26 17:32:42.714772 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:42.714754    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerDied","Data":"ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1"}
May 26 17:32:43.719759 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:43.719720    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerStarted","Data":"43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862"}
May 26 17:32:48.739322 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:48.739283    2571 generic.go:358] "Generic (PLEG): container finished" podID="248b4148-0459-48bc-a321-917204ebc623" containerID="43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862" exitCode=0
May 26 17:32:48.739802 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:48.739347    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerDied","Data":"43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862"}
May 26 17:32:49.745689 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:49.745646    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerStarted","Data":"4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659"}
May 26 17:32:49.746189 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:49.745697    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerStarted","Data":"5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271"}
May 26 17:32:49.746189 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:49.745711    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerStarted","Data":"28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37"}
May 26 17:32:49.772565 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:49.772397    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podStartSLOduration=12.772372926 podStartE2EDuration="12.772372926s" podCreationTimestamp="2026-05-26 17:32:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:32:49.770118445 +0000 UTC m=+1710.495207961" watchObservedRunningTime="2026-05-26 17:32:49.772372926 +0000 UTC m=+1710.497462427"
May 26 17:32:52.466030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:52.465981    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:52.466694 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:52.466057    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:52.466694 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:52.466075    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:52.468096 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:52.468065    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.43:8816: connect: connection refused"
May 26 17:32:52.468246 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:52.468107    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.43:8443: connect: connection refused"
May 26 17:32:52.468246 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:52.468115    2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="online" probeResult="failure" output="Get \"https://10.134.0.43:6567/health\": dial tcp 10.134.0.43:6567: connect: connection refused"
May 26 17:32:55.467330 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:55.467291    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:55.467778 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:55.467425    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:55.467949 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:55.467860    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:55.468069 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:55.467961    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:55.468139 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:55.468121    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:55.468192 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:55.468181    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:55.471751 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:55.471734    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:55.768944 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:55.768915    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:32:55.771960 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:32:55.771936    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:33:58.025495 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.025459    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"]
May 26 17:33:58.026108 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.025874    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="online" containerID="cri-o://28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37" gracePeriod=30
May 26 17:33:58.026108 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.025987    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="ui" containerID="cri-o://4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659" gracePeriod=30
May 26 17:33:58.026108 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.026062    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="offline" containerID="cri-o://5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271" gracePeriod=30
May 26 17:33:58.169372 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.169329    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"]
May 26 17:33:58.169721 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.169696    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="registry" containerID="cri-o://c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039" gracePeriod=30
May 26 17:33:58.169866 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.169840    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="online" containerID="cri-o://6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f" gracePeriod=30
May 26 17:33:58.170037 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.169840    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="offline" containerID="cri-o://67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e" gracePeriod=30
May 26 17:33:58.170037 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.169923    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="ui" containerID="cri-o://88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49" gracePeriod=30
May 26 17:33:58.968586 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.968548    2571 generic.go:358] "Generic (PLEG): container finished" podID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerID="88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49" exitCode=0
May 26 17:33:58.968586 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.968574    2571 generic.go:358] "Generic (PLEG): container finished" podID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerID="6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f" exitCode=0
May 26 17:33:58.968825 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.968611    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerDied","Data":"88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49"}
May 26 17:33:58.968825 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.968645    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerDied","Data":"6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f"}
May 26 17:33:58.970945 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.970925    2571 generic.go:358] "Generic (PLEG): container finished" podID="248b4148-0459-48bc-a321-917204ebc623" containerID="4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659" exitCode=0
May 26 17:33:58.970945 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.970941    2571 generic.go:358] "Generic (PLEG): container finished" podID="248b4148-0459-48bc-a321-917204ebc623" containerID="28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37" exitCode=0
May 26 17:33:58.971115 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.970966    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerDied","Data":"4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659"}
May 26 17:33:58.971115 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:33:58.970985    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerDied","Data":"28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37"}
May 26 17:34:05.467030 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:05.466967    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.42:8443: connect: connection refused"
May 26 17:34:05.468250 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:05.468210    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.43:8443: connect: connection refused"
May 26 17:34:05.492893 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:05.492853    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.42:6567/health\": dial tcp 10.134.0.42:6567: connect: connection refused"
May 26 17:34:05.769237 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:05.769196    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="online" probeResult="failure" output="Get \"https://10.134.0.43:6567/health\": dial tcp 10.134.0.43:6567: connect: connection refused"
May 26 17:34:15.467505 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:15.467398    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.42:8443: connect: connection refused"
May 26 17:34:15.467948 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:15.467927    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.43:8443: connect: connection refused"
May 26 17:34:15.493257 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:15.493227    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.42:6567/health\": dial tcp 10.134.0.42:6567: connect: connection refused"
May 26 17:34:15.769706 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:15.769663    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="online" probeResult="failure" output="Get \"https://10.134.0.43:6567/health\": dial tcp 10.134.0.43:6567: connect: connection refused"
May 26 17:34:19.858804 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:19.858775    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:34:19.863886 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:19.863867    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:34:25.467962 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:25.467879    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.42:8443: connect: connection refused"
May 26 17:34:25.468467 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:25.467986    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.43:8443: connect: connection refused"
May 26 17:34:25.468467 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:25.468207    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:34:25.468467 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:25.468281    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:34:25.492786 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:25.492742    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.42:6567/health\": dial tcp 10.134.0.42:6567: connect: connection refused"
May 26 17:34:25.492960 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:25.492860    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:34:25.770103 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:25.770064    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="online" probeResult="failure" output="Get \"https://10.134.0.43:6567/health\": dial tcp 10.134.0.43:6567: connect: connection refused"
May 26 17:34:25.770284 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:25.770177    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:34:28.676315 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.676294    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:34:28.765776 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.765479    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcwgl\" (UniqueName: \"kubernetes.io/projected/248b4148-0459-48bc-a321-917204ebc623-kube-api-access-kcwgl\") pod \"248b4148-0459-48bc-a321-917204ebc623\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") "
May 26 17:34:28.765776 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.765540    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-online-tls\") pod \"248b4148-0459-48bc-a321-917204ebc623\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") "
May 26 17:34:28.765776 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.765615    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/248b4148-0459-48bc-a321-917204ebc623-feast-data\") pod \"248b4148-0459-48bc-a321-917204ebc623\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") "
May 26 17:34:28.765776 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.765640    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-offline-tls\") pod \"248b4148-0459-48bc-a321-917204ebc623\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") "
May 26 17:34:28.765776 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.765672    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-ui-tls\") pod \"248b4148-0459-48bc-a321-917204ebc623\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") "
May 26 17:34:28.765776 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.765718    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/248b4148-0459-48bc-a321-917204ebc623-registry-tls\") pod \"248b4148-0459-48bc-a321-917204ebc623\" (UID: \"248b4148-0459-48bc-a321-917204ebc623\") "
May 26 17:34:28.766862 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.766254    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248b4148-0459-48bc-a321-917204ebc623-feast-data" (OuterVolumeSpecName: "feast-data") pod "248b4148-0459-48bc-a321-917204ebc623" (UID: "248b4148-0459-48bc-a321-917204ebc623"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:34:28.766862 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.766608    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248b4148-0459-48bc-a321-917204ebc623-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "248b4148-0459-48bc-a321-917204ebc623" (UID: "248b4148-0459-48bc-a321-917204ebc623"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:34:28.766862 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.766802    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/248b4148-0459-48bc-a321-917204ebc623-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:28.766862 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.766826    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/248b4148-0459-48bc-a321-917204ebc623-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:28.768407 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.768379    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "248b4148-0459-48bc-a321-917204ebc623" (UID: "248b4148-0459-48bc-a321-917204ebc623"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:34:28.768511 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.768424    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-online-tls" (OuterVolumeSpecName: "online-tls") pod "248b4148-0459-48bc-a321-917204ebc623" (UID: "248b4148-0459-48bc-a321-917204ebc623"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:34:28.768551 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.768535    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248b4148-0459-48bc-a321-917204ebc623-kube-api-access-kcwgl" (OuterVolumeSpecName: "kube-api-access-kcwgl") pod "248b4148-0459-48bc-a321-917204ebc623" (UID: "248b4148-0459-48bc-a321-917204ebc623"). InnerVolumeSpecName "kube-api-access-kcwgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:34:28.768733 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.768718    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "248b4148-0459-48bc-a321-917204ebc623" (UID: "248b4148-0459-48bc-a321-917204ebc623"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:34:28.817936 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.817914    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:34:28.867431 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.867395    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcwgl\" (UniqueName: \"kubernetes.io/projected/248b4148-0459-48bc-a321-917204ebc623-kube-api-access-kcwgl\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:28.867431 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.867426    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:28.867431 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.867440    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:28.867751 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.867451    2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/248b4148-0459-48bc-a321-917204ebc623-ui-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:28.968245 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.968213    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-online-tls\") pod \"95e1ea7e-80d5-493a-b114-e452525d55a7\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") "
May 26 17:34:28.968456 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.968262    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-offline-tls\") pod \"95e1ea7e-80d5-493a-b114-e452525d55a7\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") "
May 26 17:34:28.968456 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.968379    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-ui-tls\") pod \"95e1ea7e-80d5-493a-b114-e452525d55a7\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") "
May 26 17:34:28.968456 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.968426    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls\") pod \"95e1ea7e-80d5-493a-b114-e452525d55a7\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") "
May 26 17:34:28.968456 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.968449    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95e1ea7e-80d5-493a-b114-e452525d55a7-feast-data\") pod \"95e1ea7e-80d5-493a-b114-e452525d55a7\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") "
May 26 17:34:28.968679 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.968489    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6cv9\" (UniqueName: \"kubernetes.io/projected/95e1ea7e-80d5-493a-b114-e452525d55a7-kube-api-access-g6cv9\") pod \"95e1ea7e-80d5-493a-b114-e452525d55a7\" (UID: \"95e1ea7e-80d5-493a-b114-e452525d55a7\") "
May 26 17:34:28.969109 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.969074    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e1ea7e-80d5-493a-b114-e452525d55a7-feast-data" (OuterVolumeSpecName: "feast-data") pod "95e1ea7e-80d5-493a-b114-e452525d55a7" (UID: "95e1ea7e-80d5-493a-b114-e452525d55a7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:34:28.970679 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.970649    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "95e1ea7e-80d5-493a-b114-e452525d55a7" (UID: "95e1ea7e-80d5-493a-b114-e452525d55a7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:34:28.970679 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.970664    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "95e1ea7e-80d5-493a-b114-e452525d55a7" (UID: "95e1ea7e-80d5-493a-b114-e452525d55a7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:34:28.970847 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.970703    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-online-tls" (OuterVolumeSpecName: "online-tls") pod "95e1ea7e-80d5-493a-b114-e452525d55a7" (UID: "95e1ea7e-80d5-493a-b114-e452525d55a7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:34:28.970889 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.970871    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "95e1ea7e-80d5-493a-b114-e452525d55a7" (UID: "95e1ea7e-80d5-493a-b114-e452525d55a7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:34:28.970928 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:28.970890    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e1ea7e-80d5-493a-b114-e452525d55a7-kube-api-access-g6cv9" (OuterVolumeSpecName: "kube-api-access-g6cv9") pod "95e1ea7e-80d5-493a-b114-e452525d55a7" (UID: "95e1ea7e-80d5-493a-b114-e452525d55a7"). InnerVolumeSpecName "kube-api-access-g6cv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:34:29.067185 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.067094    2571 generic.go:358] "Generic (PLEG): container finished" podID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerID="67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e" exitCode=137
May 26 17:34:29.067185 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.067122    2571 generic.go:358] "Generic (PLEG): container finished" podID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerID="c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039" exitCode=137
May 26 17:34:29.067391 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.067184    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"
May 26 17:34:29.067391 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.067182    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerDied","Data":"67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e"}
May 26 17:34:29.067391 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.067223    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerDied","Data":"c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039"}
May 26 17:34:29.067391 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.067237    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq" event={"ID":"95e1ea7e-80d5-493a-b114-e452525d55a7","Type":"ContainerDied","Data":"f2c52c0f3ba50b13ea95d184d87da54560d946cdec75217e18d19af7d8c28aed"}
May 26 17:34:29.067391 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.067257    2571 scope.go:117] "RemoveContainer" containerID="88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49"
May 26 17:34:29.069191 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069169    2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-online-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:29.069191 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069191    2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-offline-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:29.069191 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069200    2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-ui-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:29.069191 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069209    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95e1ea7e-80d5-493a-b114-e452525d55a7-registry-tls\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:29.069191 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069219    2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95e1ea7e-80d5-493a-b114-e452525d55a7-feast-data\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:29.069191 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069233    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6cv9\" (UniqueName: \"kubernetes.io/projected/95e1ea7e-80d5-493a-b114-e452525d55a7-kube-api-access-g6cv9\") on node \"ip-10-0-129-109.ec2.internal\" DevicePath \"\""
May 26 17:34:29.069790 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069765    2571 generic.go:358] "Generic (PLEG): container finished" podID="248b4148-0459-48bc-a321-917204ebc623" containerID="5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271" exitCode=137
May 26 17:34:29.069835 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069803    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerDied","Data":"5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271"}
May 26 17:34:29.069835 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069828    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj" event={"ID":"248b4148-0459-48bc-a321-917204ebc623","Type":"ContainerDied","Data":"8238a7f1d6a7a2e7cb027a9d5c8e23ff36771df8830ce0271c1a54fda3ef99e0"}
May 26 17:34:29.069916 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.069852    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"
May 26 17:34:29.077530 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.077508    2571 scope.go:117] "RemoveContainer" containerID="67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e"
May 26 17:34:29.085825 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.085807    2571 scope.go:117] "RemoveContainer" containerID="6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f"
May 26 17:34:29.093839 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.093820    2571 scope.go:117] "RemoveContainer" containerID="c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039"
May 26 17:34:29.094417 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.094392    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"]
May 26 17:34:29.099929 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.099896    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gg4kq"]
May 26 17:34:29.101809 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.101790    2571 scope.go:117] "RemoveContainer" containerID="a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70"
May 26 17:34:29.109413 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.109391    2571 scope.go:117] "RemoveContainer" containerID="1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2"
May 26 17:34:29.111796 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.111774    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"]
May 26 17:34:29.117442 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.117421    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-j62cj"]
May 26 17:34:29.118002 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.117988    2571 scope.go:117] "RemoveContainer" containerID="88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49"
May 26 17:34:29.118307 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.118287    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49\": container with ID starting with 88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49 not found: ID does not exist" containerID="88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49"
May 26 17:34:29.118392 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.118320    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49"} err="failed to get container status \"88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49\": rpc error: code = NotFound desc = could not find container \"88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49\": container with ID starting with 88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49 not found: ID does not exist"
May 26 17:34:29.118392 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.118347    2571 scope.go:117] "RemoveContainer" containerID="67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e"
May 26 17:34:29.118603 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.118587    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e\": container with ID starting with 67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e not found: ID does not exist" containerID="67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e"
May 26 17:34:29.118639 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.118610    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e"} err="failed to get container status \"67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e\": rpc error: code = NotFound desc = could not find container \"67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e\": container with ID starting with 67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e not found: ID does not exist"
May 26 17:34:29.118639 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.118628    2571 scope.go:117] "RemoveContainer" containerID="6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f"
May 26 17:34:29.118839 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.118820    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f\": container with ID starting with 6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f not found: ID does not exist" containerID="6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f"
May 26 17:34:29.118903 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.118848    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f"} err="failed to get container status \"6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f\": rpc error: code = NotFound desc = could not find container \"6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f\": container with ID starting with 6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f not found: ID does not exist"
May 26 17:34:29.118903 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.118872    2571 scope.go:117] "RemoveContainer" containerID="c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039"
May 26 17:34:29.119126 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.119109    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039\": container with ID starting with c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039 not found: ID does not exist" containerID="c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039"
May 26 17:34:29.119174 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119132    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039"} err="failed to get container status \"c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039\": rpc error: code = NotFound desc = could not find container \"c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039\": container with ID starting with c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039 not found: ID does not exist"
May 26 17:34:29.119174 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119146    2571 scope.go:117] "RemoveContainer" containerID="a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70"
May 26 17:34:29.119342 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.119324    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70\": container with ID starting with a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70 not found: ID does not exist" containerID="a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70"
May 26 17:34:29.119403 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119349    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70"} err="failed to get container status \"a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70\": rpc error: code = NotFound desc = could not find container \"a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70\": container with ID starting with a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70 not found: ID does not exist"
May 26 17:34:29.119403 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119369    2571 scope.go:117] "RemoveContainer" containerID="1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2"
May 26 17:34:29.119567 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.119553    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2\": container with ID starting with 1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2 not found: ID does not exist" containerID="1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2"
May 26 17:34:29.119610 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119571    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2"} err="failed to get container status \"1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2\": rpc error: code = NotFound desc = could not find container \"1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2\": container with ID starting with 1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2 not found: ID does not exist"
May 26 17:34:29.119610 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119583    2571 scope.go:117] "RemoveContainer" containerID="88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49"
May 26 17:34:29.119802 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119782    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49"} err="failed to get container status \"88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49\": rpc error: code = NotFound desc = could not find container \"88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49\": container with ID starting with 88848ef5fababce0a6ec2cc3452dfc99e10d417d96bc7e17f83bbd14cbdc3e49 not found: ID does not exist"
May 26 17:34:29.119850 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119804    2571 scope.go:117] "RemoveContainer" containerID="67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e"
May 26 17:34:29.119996 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119977    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e"} err="failed to get container status \"67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e\": rpc error: code = NotFound desc = could not find container \"67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e\": container with ID starting with 67dab75d7276371017bfc2888c271709ad1e5264ce2ca1ab55d0eb5052e7960e not found: ID does not exist"
May 26 17:34:29.120163 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.119996    2571 scope.go:117] "RemoveContainer" containerID="6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f"
May 26 17:34:29.120208 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.120186    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f"} err="failed to get container status \"6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f\": rpc error: code = NotFound desc = could not find container \"6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f\": container with ID starting with 6b1cf232c0ee52b2cc96a4ae3060054618edcccb4b84ae993069b7d1046c1c7f not found: ID does not exist"
May 26 17:34:29.120208 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.120199    2571 scope.go:117] "RemoveContainer" containerID="c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039"
May 26 17:34:29.120435 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.120415    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039"} err="failed to get container status \"c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039\": rpc error: code = NotFound desc = could not find container \"c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039\": container with ID starting with c133142fdd25e89df87eba76fdc401c7ec82a28ae4bb2602bb66fcefa4f2c039 not found: ID does not exist"
May 26 17:34:29.120478 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.120438    2571 scope.go:117] "RemoveContainer" containerID="a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70"
May 26 17:34:29.120644 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.120627    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70"} err="failed to get container status \"a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70\": rpc error: code = NotFound desc = could not find container \"a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70\": container with ID starting with a53d04ee725752349a489b0dbf756dbf04fff05f549c8dc660b9701911fa8f70 not found: ID does not exist"
May 26 17:34:29.120733 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.120647    2571 scope.go:117] "RemoveContainer" containerID="1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2"
May 26 17:34:29.120834 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.120818    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2"} err="failed to get container status \"1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2\": rpc error: code = NotFound desc = could not find container \"1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2\": container with ID starting with 1135bfc4a3a2d743c8ccddb13a9b86e53a1c1db4279a0ba3e3d250da3e6e68c2 not found: ID does not exist"
May 26 17:34:29.120875 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.120834    2571 scope.go:117] "RemoveContainer" containerID="4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659"
May 26 17:34:29.128245 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.128229    2571 scope.go:117] "RemoveContainer" containerID="5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271"
May 26 17:34:29.135662 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.135644    2571 scope.go:117] "RemoveContainer" containerID="28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37"
May 26 17:34:29.142779 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.142762    2571 scope.go:117] "RemoveContainer" containerID="43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862"
May 26 17:34:29.150178 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.150159    2571 scope.go:117] "RemoveContainer" containerID="ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1"
May 26 17:34:29.157698 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.157679    2571 scope.go:117] "RemoveContainer" containerID="4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659"
May 26 17:34:29.157971 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.157950    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659\": container with ID starting with 4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659 not found: ID does not exist" containerID="4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659"
May 26 17:34:29.158049 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.157978    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659"} err="failed to get container status \"4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659\": rpc error: code = NotFound desc = could not find container \"4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659\": container with ID starting with 4449c300a3ead0e8465c8ae0c1d5570015eccc045803d99431203d7a3413d659 not found: ID does not exist"
May 26 17:34:29.158049 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.157998    2571 scope.go:117] "RemoveContainer" containerID="5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271"
May 26 17:34:29.158260 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.158239    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271\": container with ID starting with 5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271 not found: ID does not exist" containerID="5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271"
May 26 17:34:29.158308 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.158267    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271"} err="failed to get container status \"5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271\": rpc error: code = NotFound desc = could not find container \"5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271\": container with ID starting with 5c2867f28cf03765f9993eafa4c1730d27a8f1aad957b9377bb9913566c46271 not found: ID does not exist"
May 26 17:34:29.158308 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.158287    2571 scope.go:117] "RemoveContainer" containerID="28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37"
May 26 17:34:29.158526 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.158507    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37\": container with ID starting with 28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37 not found: ID does not exist" containerID="28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37"
May 26 17:34:29.158574 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.158530    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37"} err="failed to get container status \"28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37\": rpc error: code = NotFound desc = could not find container \"28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37\": container with ID starting with 28bc07059568f23936f3855eef1d3aabf746ae168706e133d00f705dda8d4e37 not found: ID does not exist"
May 26 17:34:29.158574 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.158543    2571 scope.go:117] "RemoveContainer" containerID="43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862"
May 26 17:34:29.158738 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.158720    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862\": container with ID starting with 43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862 not found: ID does not exist" containerID="43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862"
May 26 17:34:29.158782 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.158742    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862"} err="failed to get container status \"43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862\": rpc error: code = NotFound desc = could not find container \"43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862\": container with ID starting with 43a28063f5cda4d633409d4b04886f4f006986f3923fe634184ddcbdd46f6862 not found: ID does not exist"
May 26 17:34:29.158782 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.158755    2571 scope.go:117] "RemoveContainer" containerID="ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1"
May 26 17:34:29.158917 ip-10-0-129-109 kubenswrapper[2571]: E0526 17:34:29.158904    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1\": container with ID starting with ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1 not found: ID does not exist" containerID="ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1"
May 26 17:34:29.158961 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.158919    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1"} err="failed to get container status \"ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1\": rpc error: code = NotFound desc = could not find container \"ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1\": container with ID starting with ef4f591b797e61b7f81212d9323720547eb800e4beade90b56affe7215db14c1 not found: ID does not exist"
May 26 17:34:29.834319 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.834287    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="248b4148-0459-48bc-a321-917204ebc623" path="/var/lib/kubelet/pods/248b4148-0459-48bc-a321-917204ebc623/volumes"
May 26 17:34:29.834823 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:34:29.834809    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" path="/var/lib/kubelet/pods/95e1ea7e-80d5-493a-b114-e452525d55a7/volumes"
May 26 17:35:04.884969 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.884931    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r9t7h/must-gather-mppwm"]
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885365    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="ui"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885390    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="ui"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885398    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="feast-init"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885403    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="feast-init"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885413    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="ui"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885419    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="ui"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885451    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="registry"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885457    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="registry"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885466    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="offline"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885472    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="offline"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885478    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="feast-apply"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885483    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="feast-apply"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885492    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="feast-apply"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885497    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="feast-apply"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885502    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="feast-init"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885509    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="feast-init"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885519    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="offline"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885525    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="offline"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885532    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="online"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885537    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="online"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885545    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="online"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885550    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="online"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885601    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="online"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885609    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="online"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885616    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="ui"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885622    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="offline"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885629    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="ui"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885635    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="95e1ea7e-80d5-493a-b114-e452525d55a7" containerName="registry"
May 26 17:35:04.887445 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.885642    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="248b4148-0459-48bc-a321-917204ebc623" containerName="offline"
May 26 17:35:04.888663 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.888649    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9t7h/must-gather-mppwm"
May 26 17:35:04.891184 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.891158    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9t7h\"/\"kube-root-ca.crt\""
May 26 17:35:04.891302 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.891210    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9t7h\"/\"openshift-service-ca.crt\""
May 26 17:35:04.892483 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.892460    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r9t7h\"/\"default-dockercfg-q9zft\""
May 26 17:35:04.895115 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.895094    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9t7h/must-gather-mppwm"]
May 26 17:35:04.968173 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.968140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c02231a-8604-4d7e-ac20-24d1f8f1b23b-must-gather-output\") pod \"must-gather-mppwm\" (UID: \"4c02231a-8604-4d7e-ac20-24d1f8f1b23b\") " pod="openshift-must-gather-r9t7h/must-gather-mppwm"
May 26 17:35:04.968173 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:04.968182    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5sx5\" (UniqueName: \"kubernetes.io/projected/4c02231a-8604-4d7e-ac20-24d1f8f1b23b-kube-api-access-l5sx5\") pod \"must-gather-mppwm\" (UID: \"4c02231a-8604-4d7e-ac20-24d1f8f1b23b\") " pod="openshift-must-gather-r9t7h/must-gather-mppwm"
May 26 17:35:05.069739 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:05.069701    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c02231a-8604-4d7e-ac20-24d1f8f1b23b-must-gather-output\") pod \"must-gather-mppwm\" (UID: \"4c02231a-8604-4d7e-ac20-24d1f8f1b23b\") " pod="openshift-must-gather-r9t7h/must-gather-mppwm"
May 26 17:35:05.069739 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:05.069749    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l5sx5\" (UniqueName: \"kubernetes.io/projected/4c02231a-8604-4d7e-ac20-24d1f8f1b23b-kube-api-access-l5sx5\") pod \"must-gather-mppwm\" (UID: \"4c02231a-8604-4d7e-ac20-24d1f8f1b23b\") " pod="openshift-must-gather-r9t7h/must-gather-mppwm"
May 26 17:35:05.070099 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:05.070075    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c02231a-8604-4d7e-ac20-24d1f8f1b23b-must-gather-output\") pod \"must-gather-mppwm\" (UID: \"4c02231a-8604-4d7e-ac20-24d1f8f1b23b\") " pod="openshift-must-gather-r9t7h/must-gather-mppwm"
May 26 17:35:05.077338 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:05.077314    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5sx5\" (UniqueName: \"kubernetes.io/projected/4c02231a-8604-4d7e-ac20-24d1f8f1b23b-kube-api-access-l5sx5\") pod \"must-gather-mppwm\" (UID: \"4c02231a-8604-4d7e-ac20-24d1f8f1b23b\") " pod="openshift-must-gather-r9t7h/must-gather-mppwm"
May 26 17:35:05.213675 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:05.213593    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9t7h/must-gather-mppwm"
May 26 17:35:05.340630 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:05.340605    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9t7h/must-gather-mppwm"]
May 26 17:35:05.343266 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:35:05.343233    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c02231a_8604_4d7e_ac20_24d1f8f1b23b.slice/crio-b10c5e0fa4272c611ad5ce5216cbf94e0e859b490a7cb7c27d377aa516d65f2b WatchSource:0}: Error finding container b10c5e0fa4272c611ad5ce5216cbf94e0e859b490a7cb7c27d377aa516d65f2b: Status 404 returned error can't find the container with id b10c5e0fa4272c611ad5ce5216cbf94e0e859b490a7cb7c27d377aa516d65f2b
May 26 17:35:06.191191 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:06.191154    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9t7h/must-gather-mppwm" event={"ID":"4c02231a-8604-4d7e-ac20-24d1f8f1b23b","Type":"ContainerStarted","Data":"b10c5e0fa4272c611ad5ce5216cbf94e0e859b490a7cb7c27d377aa516d65f2b"}
May 26 17:35:07.196539 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:07.196498    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9t7h/must-gather-mppwm" event={"ID":"4c02231a-8604-4d7e-ac20-24d1f8f1b23b","Type":"ContainerStarted","Data":"f45324b74f39d83408fa010b460d6dfdea0aa5eb78f5b11636e2889b9e32f23a"}
May 26 17:35:07.196539 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:07.196544    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9t7h/must-gather-mppwm" event={"ID":"4c02231a-8604-4d7e-ac20-24d1f8f1b23b","Type":"ContainerStarted","Data":"96ce6aa6303aada16f4f2fda3840871bfb228f736f4cd064b3962bf357e819de"}
May 26 17:35:07.213450 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:07.213354    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-r9t7h/must-gather-mppwm" podStartSLOduration=2.42232556 podStartE2EDuration="3.213334862s" podCreationTimestamp="2026-05-26 17:35:04 +0000 UTC" firstStartedPulling="2026-05-26 17:35:05.345083992 +0000 UTC m=+1846.070173480" lastFinishedPulling="2026-05-26 17:35:06.136093301 +0000 UTC m=+1846.861182782" observedRunningTime="2026-05-26 17:35:07.211081666 +0000 UTC m=+1847.936171167" watchObservedRunningTime="2026-05-26 17:35:07.213334862 +0000 UTC m=+1847.938424362"
May 26 17:35:07.365644 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:07.365613    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2xt7z_7e2672b8-f37b-4393-8b5d-2c9dd96bb85f/global-pull-secret-syncer/0.log"
May 26 17:35:07.520405 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:07.520353    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r6jrs_8ce63184-976c-4e94-9b6c-c94a79abbf53/konnectivity-agent/0.log"
May 26 17:35:07.564967 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:07.564942    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-109.ec2.internal_f09d26b0580610a005e903ab2681d50d/haproxy/0.log"
May 26 17:35:11.090091 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.090052    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-99qnk_38027300-8fba-4a1d-8f9c-745e9942954a/cluster-monitoring-operator/0.log"
May 26 17:35:11.123073 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.123038    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-mfcrs_ae21fdd2-8056-4641-9dd2-d0ef955e4a0a/kube-state-metrics/0.log"
May 26 17:35:11.148939 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.148909    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-mfcrs_ae21fdd2-8056-4641-9dd2-d0ef955e4a0a/kube-rbac-proxy-main/0.log"
May 26 17:35:11.170027 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.169947    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-mfcrs_ae21fdd2-8056-4641-9dd2-d0ef955e4a0a/kube-rbac-proxy-self/0.log"
May 26 17:35:11.197006 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.196972    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d8d68d57b-m8qpx_8ef69f78-f149-4702-b560-d1a103725d7b/metrics-server/0.log"
May 26 17:35:11.221853 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.221821    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-d5zwx_380972e0-8864-4ad5-8e16-85ce0e8a34b2/monitoring-plugin/0.log"
May 26 17:35:11.368916 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.368828    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrhrc_08843935-911b-4d43-ad28-f7c15f527321/node-exporter/0.log"
May 26 17:35:11.388043 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.388001    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrhrc_08843935-911b-4d43-ad28-f7c15f527321/kube-rbac-proxy/0.log"
May 26 17:35:11.409895 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.409866    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrhrc_08843935-911b-4d43-ad28-f7c15f527321/init-textfile/0.log"
May 26 17:35:11.438901 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.438872    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-8fhl2_089437a0-9dc8-4425-9660-bbbc7e10ab68/kube-rbac-proxy-main/0.log"
May 26 17:35:11.457805 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.457774    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-8fhl2_089437a0-9dc8-4425-9660-bbbc7e10ab68/kube-rbac-proxy-self/0.log"
May 26 17:35:11.477689 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.477658    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-8fhl2_089437a0-9dc8-4425-9660-bbbc7e10ab68/openshift-state-metrics/0.log"
May 26 17:35:11.671177 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.671102    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-mlxhs_d9d4b70a-fb28-4c67-a645-07307550e2db/prometheus-operator-admission-webhook/0.log"
May 26 17:35:11.702229 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.702194    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bd9755968-2d2zx_6381a615-e822-4c85-89cc-fd790ef6a0da/telemeter-client/0.log"
May 26 17:35:11.721637 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.721608    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bd9755968-2d2zx_6381a615-e822-4c85-89cc-fd790ef6a0da/reload/0.log"
May 26 17:35:11.747861 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.747829    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bd9755968-2d2zx_6381a615-e822-4c85-89cc-fd790ef6a0da/kube-rbac-proxy/0.log"
May 26 17:35:11.789354 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.789322    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc4d8cdc8-p7m7f_e77b3862-aab3-485a-8910-bfa16b5c9050/thanos-query/0.log"
May 26 17:35:11.817642 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.817616    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc4d8cdc8-p7m7f_e77b3862-aab3-485a-8910-bfa16b5c9050/kube-rbac-proxy-web/0.log"
May 26 17:35:11.851634 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.851607    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc4d8cdc8-p7m7f_e77b3862-aab3-485a-8910-bfa16b5c9050/kube-rbac-proxy/0.log"
May 26 17:35:11.876379 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.876353    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc4d8cdc8-p7m7f_e77b3862-aab3-485a-8910-bfa16b5c9050/prom-label-proxy/0.log"
May 26 17:35:11.903382 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.903352    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc4d8cdc8-p7m7f_e77b3862-aab3-485a-8910-bfa16b5c9050/kube-rbac-proxy-rules/0.log"
May 26 17:35:11.935808 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:11.935677    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc4d8cdc8-p7m7f_e77b3862-aab3-485a-8910-bfa16b5c9050/kube-rbac-proxy-metrics/0.log"
May 26 17:35:13.670883 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:13.670853    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-dvsmd_27ace5ae-76be-4c79-93ce-e4095cb098ab/download-server/0.log"
May 26 17:35:13.993341 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:13.993306    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-pg9gb_1f1f5c29-30bc-48b5-96b8-2d130fa358c1/volume-data-source-validator/0.log"
May 26 17:35:14.397761 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.397670    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"]
May 26 17:35:14.403136 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.403105    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.410165 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.410138    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"]
May 26 17:35:14.465560 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.465520    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-sys\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.465916 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.465896    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-lib-modules\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.466135 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.466108    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-proc\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.466224 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.466208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-podres\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.466294 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.466241    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv47b\" (UniqueName: \"kubernetes.io/projected/2695cd9c-c6e1-49d5-9984-ca6105edbe28-kube-api-access-wv47b\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.567525 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.567490    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-proc\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.567690 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.567544    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-podres\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.567690 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.567661    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-proc\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.567799 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.567724    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wv47b\" (UniqueName: \"kubernetes.io/projected/2695cd9c-c6e1-49d5-9984-ca6105edbe28-kube-api-access-wv47b\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.567799 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.567743    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-podres\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.567908 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.567805    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-sys\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.567908 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.567857    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-lib-modules\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.568060 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.567928    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-sys\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.568060 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.567980    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2695cd9c-c6e1-49d5-9984-ca6105edbe28-lib-modules\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.568803 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.568782    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4c8fb_91c3c9fd-a70c-4e23-85db-d5544f5c9a73/dns/0.log"
May 26 17:35:14.575568 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.575548    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv47b\" (UniqueName: \"kubernetes.io/projected/2695cd9c-c6e1-49d5-9984-ca6105edbe28-kube-api-access-wv47b\") pod \"perf-node-gather-daemonset-5hpzn\" (UID: \"2695cd9c-c6e1-49d5-9984-ca6105edbe28\") " pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.587979 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.587962    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4c8fb_91c3c9fd-a70c-4e23-85db-d5544f5c9a73/kube-rbac-proxy/0.log"
May 26 17:35:14.709560 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.709477    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m97tn_87203750-2eb1-4b8c-9c94-b7ca88363b40/dns-node-resolver/0.log"
May 26 17:35:14.716998 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.716977    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:14.866528 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:14.866500    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"]
May 26 17:35:14.868878 ip-10-0-129-109 kubenswrapper[2571]: W0526 17:35:14.868848    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2695cd9c_c6e1_49d5_9984_ca6105edbe28.slice/crio-547128f3787d19a9c1a0142b298e8d13c8e05f7a32825db4dc1dabafe5d614be WatchSource:0}: Error finding container 547128f3787d19a9c1a0142b298e8d13c8e05f7a32825db4dc1dabafe5d614be: Status 404 returned error can't find the container with id 547128f3787d19a9c1a0142b298e8d13c8e05f7a32825db4dc1dabafe5d614be
May 26 17:35:15.114892 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:15.114862    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6w2fz_f9c75f58-61b8-4762-a4b4-ceef1c52bf7b/node-ca/0.log"
May 26 17:35:15.246731 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:15.246701    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn" event={"ID":"2695cd9c-c6e1-49d5-9984-ca6105edbe28","Type":"ContainerStarted","Data":"3da901db565b7218395672afd3312a727734e20b3b6c191ac864aa731faac920"}
May 26 17:35:15.246911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:15.246738    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn" event={"ID":"2695cd9c-c6e1-49d5-9984-ca6105edbe28","Type":"ContainerStarted","Data":"547128f3787d19a9c1a0142b298e8d13c8e05f7a32825db4dc1dabafe5d614be"}
May 26 17:35:15.246911 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:15.246779    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:15.267269 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:15.262642    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn" podStartSLOduration=1.26261855 podStartE2EDuration="1.26261855s" podCreationTimestamp="2026-05-26 17:35:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:35:15.261683856 +0000 UTC m=+1855.986773358" watchObservedRunningTime="2026-05-26 17:35:15.26261855 +0000 UTC m=+1855.987708052"
May 26 17:35:15.745536 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:15.745501    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-978fdbd8-97ct5_9008dab9-05e5-43b6-bdc8-077476da5900/router/0.log"
May 26 17:35:16.049354 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:16.049279    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mvfcs_b5935fea-b672-471d-a707-a16fc6f3203e/serve-healthcheck-canary/0.log"
May 26 17:35:16.341700 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:16.341618    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-w47qs_5f96e4a7-3baa-47a8-b688-ed682aaeba85/insights-operator/0.log"
May 26 17:35:16.342406 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:16.342381    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-w47qs_5f96e4a7-3baa-47a8-b688-ed682aaeba85/insights-operator/1.log"
May 26 17:35:16.473216 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:16.473189    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tk2sx_2fb0788f-2272-49e4-8f3f-ca83c46f2004/kube-rbac-proxy/0.log"
May 26 17:35:16.493899 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:16.493870    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tk2sx_2fb0788f-2272-49e4-8f3f-ca83c46f2004/exporter/0.log"
May 26 17:35:16.513600 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:16.513563    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tk2sx_2fb0788f-2272-49e4-8f3f-ca83c46f2004/extractor/0.log"
May 26 17:35:20.199966 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:20.199886    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-2d4lx_dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27/migrator/0.log"
May 26 17:35:20.217297 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:20.217271    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-2d4lx_dcdfbabf-4c73-46f3-a1f6-bf7cbffbbe27/graceful-termination/0.log"
May 26 17:35:20.460985 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:20.460900    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-gbm7m_df3cb020-a879-45c7-8fff-b38a4b5ccf30/kube-storage-version-migrator-operator/1.log"
May 26 17:35:20.462501 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:20.462478    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-gbm7m_df3cb020-a879-45c7-8fff-b38a4b5ccf30/kube-storage-version-migrator-operator/0.log"
May 26 17:35:21.182814 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.182786    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qh7j_a60644b9-b57b-4023-bc10-ddb686016835/kube-multus-additional-cni-plugins/0.log"
May 26 17:35:21.206636 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.206603    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qh7j_a60644b9-b57b-4023-bc10-ddb686016835/egress-router-binary-copy/0.log"
May 26 17:35:21.224696 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.224660    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qh7j_a60644b9-b57b-4023-bc10-ddb686016835/cni-plugins/0.log"
May 26 17:35:21.242540 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.242513    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qh7j_a60644b9-b57b-4023-bc10-ddb686016835/bond-cni-plugin/0.log"
May 26 17:35:21.260630 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.260602    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qh7j_a60644b9-b57b-4023-bc10-ddb686016835/routeoverride-cni/0.log"
May 26 17:35:21.261351 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.261323    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r9t7h/perf-node-gather-daemonset-5hpzn"
May 26 17:35:21.284558 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.284533    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qh7j_a60644b9-b57b-4023-bc10-ddb686016835/whereabouts-cni-bincopy/0.log"
May 26 17:35:21.304274 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.304248    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8qh7j_a60644b9-b57b-4023-bc10-ddb686016835/whereabouts-cni/0.log"
May 26 17:35:21.612899 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.612854    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xrkfv_b89e58ac-045c-4d32-951c-a0a00befc975/kube-multus/0.log"
May 26 17:35:21.707571 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.707541    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lztjw_a13dbe45-e72d-4315-a2a1-c4736202fd9b/network-metrics-daemon/0.log"
May 26 17:35:21.722852 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:21.722827    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lztjw_a13dbe45-e72d-4315-a2a1-c4736202fd9b/kube-rbac-proxy/0.log"
May 26 17:35:22.591442 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:22.591409    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-controller/0.log"
May 26 17:35:22.608506 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:22.608480    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/0.log"
May 26 17:35:22.625075 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:22.625042    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovn-acl-logging/1.log"
May 26 17:35:22.647760 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:22.647731    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/kube-rbac-proxy-node/0.log"
May 26 17:35:22.670367 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:22.670337    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/kube-rbac-proxy-ovn-metrics/0.log"
May 26 17:35:22.688646 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:22.688601    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/northd/0.log"
May 26 17:35:22.712539 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:22.712516    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/nbdb/0.log"
May 26 17:35:22.735358 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:22.735327    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/sbdb/0.log"
May 26 17:35:22.920272 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:22.920180    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r56x2_2753c993-0102-4e57-b557-f03934174f89/ovnkube-controller/0.log"
May 26 17:35:24.208323 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:24.208294    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-gzk92_7c05bfdb-3638-40a7-89a5-ab54cb3a6299/check-endpoints/0.log"
May 26 17:35:24.245794 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:24.245761    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fjgrn_9bd326f5-4fac-48da-b113-83b2234bc506/network-check-target-container/0.log"
May 26 17:35:25.130101 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:25.129793    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-b498j_0ace1a7d-bd84-4be0-8dc5-63f855978478/iptables-alerter/0.log"
May 26 17:35:25.715197 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:25.715164    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-bz4lz_c9b5a4af-6d8b-411b-8339-65a20f76591c/tuned/0.log"
May 26 17:35:27.191766 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:27.191730    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-cvqdr_937078be-09eb-4f20-9ef7-eab92c00b8bd/cluster-samples-operator/0.log"
May 26 17:35:27.220574 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:27.220545    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-cvqdr_937078be-09eb-4f20-9ef7-eab92c00b8bd/cluster-samples-operator-watch/0.log"
May 26 17:35:28.073631 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:28.073598    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-kkdjh_a754be44-8031-4393-ae82-46994bc250ae/service-ca-operator/1.log"
May 26 17:35:28.075251 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:28.075221    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-kkdjh_a754be44-8031-4393-ae82-46994bc250ae/service-ca-operator/0.log"
May 26 17:35:28.756341 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:28.756303    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-dvh4h_2067d39e-24b4-4da4-9631-2f927333a9ef/csi-driver/0.log"
May 26 17:35:28.776038 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:28.775983    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-dvh4h_2067d39e-24b4-4da4-9631-2f927333a9ef/csi-node-driver-registrar/0.log"
May 26 17:35:28.793414 ip-10-0-129-109 kubenswrapper[2571]: I0526 17:35:28.793386    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-dvh4h_2067d39e-24b4-4da4-9631-2f927333a9ef/csi-liveness-probe/0.log"