May 29 01:58:23.298024 ip-10-0-129-62 systemd[1]: Starting Kubernetes Kubelet...
May 29 01:58:23.716077 ip-10-0-129-62 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 01:58:23.716732 ip-10-0-129-62 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 29 01:58:23.716732 ip-10-0-129-62 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 01:58:23.716732 ip-10-0-129-62 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 29 01:58:23.716732 ip-10-0-129-62 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 01:58:23.717104 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.717017    2580 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 29 01:58:23.720090 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720070    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 01:58:23.720090 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720086    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 01:58:23.720090 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720091    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 01:58:23.720090 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720097    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720101    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720106    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720110    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720114    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720118    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720121    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720125    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720129    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720132    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720136    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720141    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720145    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720149    2580 feature_gate.go:328] unrecognized feature gate: Example
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720153    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720156    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720160    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720164    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720168    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720181    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 01:58:23.720327 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720186    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720189    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720193    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720198    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720202    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720206    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720209    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720213    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720218    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720222    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720226    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720230    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720235    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720239    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720244    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720253    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720260    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720265    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720269    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720275    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 01:58:23.721138 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720280    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720284    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720288    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720292    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720296    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720301    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720305    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720309    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720313    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720317    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720322    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720325    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720330    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720334    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720338    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720343    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720347    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720351    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720356    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720360    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 01:58:23.722040 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720364    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720368    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720373    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720377    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720382    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720386    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720392    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720406    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720410    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720414    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720419    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720423    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720430    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720435    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720440    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720444    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720448    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720455    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720461    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 01:58:23.722905 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720465    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720470    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720475    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.720479    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721141    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721149    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721154    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721160    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721165    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721170    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721175    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721180    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721185    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721189    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721193    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721197    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721201    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721205    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721209    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721214    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 01:58:23.723548 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721219    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721223    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721227    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721231    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721236    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721245    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721249    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721253    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721258    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721264    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721268    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721272    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721276    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721280    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721284    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721289    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721293    2580 feature_gate.go:328] unrecognized feature gate: Example
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721301    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721306    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721310    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 01:58:23.724171 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721315    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721319    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721324    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721328    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721333    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721337    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721341    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721345    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721350    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721354    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721358    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721362    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721367    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721371    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721375    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721379    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721383    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721388    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721400    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 01:58:23.724817 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721405    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721409    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721415    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721420    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721424    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721429    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721433    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721437    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721440    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721444    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721449    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721452    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721457    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721461    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721465    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721469    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721473    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721477    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721481    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 01:58:23.725283 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721485    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721490    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721495    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721499    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721503    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721511    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721516    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721520    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721524    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721528    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721532    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.721536    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721660    2580 flags.go:64] FLAG: --address="0.0.0.0"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721672    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721692    2580 flags.go:64] FLAG: --anonymous-auth="true"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721698    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721706    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721712    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721719    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721730    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721735    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721740    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 29 01:58:23.725860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721746    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721751    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721756    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721761    2580 flags.go:64] FLAG: --cgroup-root=""
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721766    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721771    2580 flags.go:64] FLAG: --client-ca-file=""
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721776    2580 flags.go:64] FLAG: --cloud-config=""
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721781    2580 flags.go:64] FLAG: --cloud-provider="external"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721790    2580 flags.go:64] FLAG: --cluster-dns="[]"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721799    2580 flags.go:64] FLAG: --cluster-domain=""
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721804    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721810    2580 flags.go:64] FLAG: --config-dir=""
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721814    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721820    2580 flags.go:64] FLAG: --container-log-max-files="5"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721826    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721833    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721839    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721845    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721850    2580 flags.go:64] FLAG: --contention-profiling="false"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721855    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721860    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721865    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721870    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721876    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721883    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 29 01:58:23.726391 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721887    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721893    2580 flags.go:64] FLAG: --enable-load-reader="false"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721898    2580 flags.go:64] FLAG: --enable-server="true"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721903    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721909    2580 flags.go:64] FLAG: --event-burst="100"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721914    2580 flags.go:64] FLAG: --event-qps="50"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721919    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721924    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721929    2580 flags.go:64] FLAG: --eviction-hard=""
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721935    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721940    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721945    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721950    2580 flags.go:64] FLAG: --eviction-soft=""
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721955    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721960    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721967    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721973    2580 flags.go:64] FLAG: --experimental-mounter-path=""
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721978    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721983    2580 flags.go:64] FLAG: --fail-swap-on="true"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721988    2580 flags.go:64] FLAG: --feature-gates=""
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721995    2580 flags.go:64] FLAG: --file-check-frequency="20s"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.721999    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722006    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722012    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722017    2580 flags.go:64] FLAG: --healthz-port="10248"
May 29 01:58:23.727045 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722022    2580 flags.go:64] FLAG: --help="false"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722052    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-129-62.ec2.internal"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722059    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722064    2580 flags.go:64] FLAG: --http-check-frequency="20s"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722069    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722076    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722081    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722087    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722092    2580 flags.go:64] FLAG: --image-service-endpoint=""
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722097    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722102    2580 flags.go:64] FLAG: --kube-api-burst="100"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722107    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722112    2580 flags.go:64] FLAG: --kube-api-qps="50"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722117    2580 flags.go:64] FLAG: --kube-reserved=""
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722122    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722126    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722132    2580 flags.go:64] FLAG: --kubelet-cgroups=""
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722136    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722141    2580 flags.go:64] FLAG: --lock-file=""
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722146    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722151    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722156    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722167    2580 flags.go:64] FLAG: --log-json-split-stream="false"
May 29 01:58:23.727667 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722172    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722177    2580 flags.go:64] FLAG: --log-text-split-stream="false"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722182    2580 flags.go:64] FLAG: --logging-format="text"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722187    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722192    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722197    2580 flags.go:64] FLAG: --manifest-url=""
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722204    2580 flags.go:64] FLAG: --manifest-url-header=""
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722211    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722216    2580 flags.go:64] FLAG: --max-open-files="1000000"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722223    2580 flags.go:64] FLAG: --max-pods="110"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722227    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722232    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722237    2580 flags.go:64] FLAG: --memory-manager-policy="None"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722243    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722248    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722252    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722258    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722278    2580 flags.go:64] FLAG: --node-status-max-images="50"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722284    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722289    2580 flags.go:64] FLAG: --oom-score-adj="-999"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722294    2580 flags.go:64] FLAG: --pod-cidr=""
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722299    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722307    2580 flags.go:64] FLAG: --pod-manifest-path=""
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722312    2580 flags.go:64] FLAG: --pod-max-pids="-1"
May 29 01:58:23.728230 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722316    2580 flags.go:64] FLAG: --pods-per-core="0"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722321    2580 flags.go:64] FLAG: --port="10250"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722327    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722332    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ff7f20805043ce8c"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722337    2580 flags.go:64] FLAG: --qos-reserved=""
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722342    2580 flags.go:64] FLAG: --read-only-port="10255"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722347    2580 flags.go:64] FLAG: --register-node="true"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722352    2580 flags.go:64] FLAG: --register-schedulable="true"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722358    2580 flags.go:64] FLAG: --register-with-taints=""
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722365    2580 flags.go:64] FLAG: --registry-burst="10"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722369    2580 flags.go:64] FLAG: --registry-qps="5"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722374    2580 flags.go:64] FLAG: --reserved-cpus=""
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722379    2580 flags.go:64] FLAG: --reserved-memory=""
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722385    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722393    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722398    2580 flags.go:64] FLAG: --rotate-certificates="false"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722403    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722408    2580 flags.go:64] FLAG: --runonce="false"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722413    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722418    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722423    2580 flags.go:64] FLAG: --seccomp-default="false"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722428    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722433    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722438    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722443    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722448    2580 flags.go:64] FLAG: --storage-driver-password="root"
May 29 01:58:23.728911 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722461    2580 flags.go:64] FLAG: --storage-driver-secure="false"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722466    2580 flags.go:64] FLAG: --storage-driver-table="stats"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722471    2580 flags.go:64] FLAG: --storage-driver-user="root"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722475    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722480    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722486    2580 flags.go:64] FLAG: --system-cgroups=""
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722490    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722499    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722504    2580 flags.go:64] FLAG: --tls-cert-file=""
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722509    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722519    2580 flags.go:64] FLAG: --tls-min-version=""
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722524    2580 flags.go:64] FLAG: --tls-private-key-file=""
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722529    2580 flags.go:64] FLAG: --topology-manager-policy="none"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722534    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722541    2580 flags.go:64] FLAG: --topology-manager-scope="container"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722546    2580 flags.go:64] FLAG: --v="2"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722553    2580 flags.go:64] FLAG: --version="false"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722559    2580 flags.go:64] FLAG: --vmodule=""
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722566    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.722572    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722755    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722763    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722771    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722777    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 01:58:23.729524 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722783    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722787    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722793    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722797    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722802    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722807    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722811    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722816    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722821    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722831    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722837    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722841    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722845    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722850    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722854    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722859    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722863    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722867    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722871    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722876    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 01:58:23.730150 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722880    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722884    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722890    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722895    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722899    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722903    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722907    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722911    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722917    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722921    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722925    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722930    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722934    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722938    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722943    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722947    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722951    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722955    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722960    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722964    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 01:58:23.730686 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722968    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722972    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722978    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722982    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722986    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722991    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722995    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.722999    2580 feature_gate.go:328] unrecognized feature gate: Example
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723004    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723008    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723012    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723016    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723021    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723025    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723031    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723035    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723039    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723044    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723048    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723052    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 01:58:23.731173 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723056    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723061    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723065    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723069    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723073    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723078    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723082    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723086    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723090    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723094    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723098    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723103    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723107    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723113    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723119    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723131    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723136    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723140    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723145    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723149    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 01:58:23.731669 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723154    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.723158    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.723166    2580 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.730538    2580 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.730555    2580 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730616    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730622    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730625    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730628    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730632    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730635    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730638    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730641    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730644    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730646    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730649    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 01:58:23.732157 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730652    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730654    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730657    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730660    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730664    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730667    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730670    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730674    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730676    2580 feature_gate.go:328] unrecognized feature gate: Example
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730679    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730682    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730684    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730687    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730690    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730693    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730696    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730699    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730701    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730704    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 01:58:23.732556 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730707    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730711    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730714    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730716    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730719    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730722    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730724    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730727    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730729    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730732    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730736    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730741    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730744    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730747    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730750    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730753    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730756    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730758    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730761    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730764    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 01:58:23.733055 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730766    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730769    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730772    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730775    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730777    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730780    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730782    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730785    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730787    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730790    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730792    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730795    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730797    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730801    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730805    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730808    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730811    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730813    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730816    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730819    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 01:58:23.733577 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730821    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730823    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730826    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730828    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730831    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730834    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730836    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730839    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730841    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730844    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730846    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730849    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730851    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730854    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730856    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730859    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 01:58:23.734086 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.730864    2580 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730960    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730965    2580 feature_gate.go:328] unrecognized feature gate: Example
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730968    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730971    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730974    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730977    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730979    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730981    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730984    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730987    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730990    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730993    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730995    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.730998    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731000    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731004    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731007    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731010    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 01:58:23.734538 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731013    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731015    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731018    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731020    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731023    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731025    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731028    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731030    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731033    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731035    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731037    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731040    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731042    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731045    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731047    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731050    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731052    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731055    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731057    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 01:58:23.735037 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731059    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731062    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731064    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731067    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731070    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731073    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731076    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731079    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731081    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731083    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731086    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731088    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731091    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731093    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731096    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731099    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731101    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731104    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731106    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731108    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 01:58:23.735505 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731111    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731113    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731116    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731118    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731121    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731123    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731126    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731128    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731131    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731134    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731136    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731139    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731142    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731146    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731149    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731152    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731155    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731158    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731161    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731164    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 01:58:23.736016 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731167    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731170    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731173    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731175    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731178    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731181    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731184    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731187    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:23.731190    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.731195    2580 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.732031    2580 server.go:962] "Client rotation is on, will bootstrap in background"
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.734985    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.735922    2580 server.go:1019] "Starting client certificate rotation"
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.736032    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 29 01:58:23.736501 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.736074    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 29 01:58:23.759974 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.759952    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 29 01:58:23.765696 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.765669    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 29 01:58:23.784640 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.784621    2580 log.go:25] "Validated CRI v1 runtime API"
May 29 01:58:23.789839 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.789823    2580 log.go:25] "Validated CRI v1 image API"
May 29 01:58:23.791732 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.791714    2580 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 29 01:58:23.796569 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.796546    2580 fs.go:135] Filesystem UUIDs: map[1ad03ccb-0cf4-4542-9627-001be9e2e7f9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c2de182e-ccca-44e4-ae03-cfa40a3ae65d:/dev/nvme0n1p4]
May 29 01:58:23.796638 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.796570    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 29 01:58:23.797458 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.797315    2580 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 29 01:58:23.801184 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.801072    2580 manager.go:217] Machine: {Timestamp:2026-05-29 01:58:23.800556647 +0000 UTC m=+0.383547542 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499998 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2724180b185a3cc24e3ba309b22332 SystemUUID:ec272418-0b18-5a3c-c24e-3ba309b22332 BootID:bc4cdc9f-c7a1-4308-b557-f42d2b5dc4f2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:84:63:97:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:84:63:97:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:89:db:eb:a4:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 29 01:58:23.801866 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.801856    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 29 01:58:23.801953 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.801941    2580 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 29 01:58:23.803869 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.803846    2580 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 29 01:58:23.804004 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.803873    2580 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-62.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 29 01:58:23.804047 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.804014    2580 topology_manager.go:138] "Creating topology manager with none policy"
May 29 01:58:23.804047 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.804023    2580 container_manager_linux.go:306] "Creating device plugin manager"
May 29 01:58:23.804047 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.804030    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 29 01:58:23.804634 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.804623    2580 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 29 01:58:23.806869 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.806860    2580 state_mem.go:36] "Initialized new in-memory state store"
May 29 01:58:23.806985 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.806976    2580 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 29 01:58:23.809569 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.809560    2580 kubelet.go:491] "Attempting to sync node with API server"
May 29 01:58:23.809623 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.809573    2580 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 29 01:58:23.809623 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.809584    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 29 01:58:23.809623 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.809605    2580 kubelet.go:397] "Adding apiserver pod source"
May 29 01:58:23.809623 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.809614    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 29 01:58:23.810701 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.810689    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 29 01:58:23.810747 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.810708    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 29 01:58:23.813531 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.813513    2580 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 29 01:58:23.817517 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.817487    2580 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 29 01:58:23.819078 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819062    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 29 01:58:23.819129 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819090    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 29 01:58:23.819129 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819099    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 29 01:58:23.819129 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819106    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 29 01:58:23.819129 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819115    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 29 01:58:23.819129 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819122    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 29 01:58:23.819129 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819129    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 29 01:58:23.819294 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819139    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 29 01:58:23.819294 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819156    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 29 01:58:23.819294 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819165    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 29 01:58:23.819294 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819189    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 29 01:58:23.819294 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.819216    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 29 01:58:23.820257 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.820242    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 29 01:58:23.820317 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.820291    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 29 01:58:23.823144 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.823120    2580 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 29 01:58:23.823144 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.823123    2580 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 29 01:58:23.824124 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.824109    2580 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 29 01:58:23.824171 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.824152    2580 server.go:1295] "Started kubelet"
May 29 01:58:23.824240 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.824210    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 29 01:58:23.824324 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.824266    2580 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 29 01:58:23.824371 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.824340    2580 server_v1.go:47] "podresources" method="list" useActivePods=true
May 29 01:58:23.825120 ip-10-0-129-62 systemd[1]: Started Kubernetes Kubelet.
May 29 01:58:23.825689 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.825672    2580 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 29 01:58:23.825881 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.825837    2580 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-62.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 29 01:58:23.827115 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.827101    2580 server.go:317] "Adding debug handlers to kubelet server"
May 29 01:58:23.835043 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.835024    2580 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 29 01:58:23.835429 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.835412    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 29 01:58:23.835580 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.834690    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-62.ec2.internal.18b3e6abec8fd87a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-62.ec2.internal,UID:ip-10-0-129-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-62.ec2.internal,},FirstTimestamp:2026-05-29 01:58:23.824124026 +0000 UTC m=+0.407114922,LastTimestamp:2026-05-29 01:58:23.824124026 +0000 UTC m=+0.407114922,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-62.ec2.internal,}"
May 29 01:58:23.835962 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.835943    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 29 01:58:23.836173 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.836158    2580 volume_manager.go:295] "The desired_state_of_world populator starts"
May 29 01:58:23.836217 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.836178    2580 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 29 01:58:23.836293 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.836277    2580 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 29 01:58:23.836366 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.836345    2580 reconstruct.go:97] "Volume reconstruction finished"
May 29 01:58:23.836366 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.836358    2580 reconciler.go:26] "Reconciler: start to sync state"
May 29 01:58:23.838237 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.836849    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 29 01:58:23.838237 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.836873    2580 factory.go:55] Registering systemd factory
May 29 01:58:23.838237 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.836882    2580 factory.go:223] Registration of the systemd container factory successfully
May 29 01:58:23.838237 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.836959    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:23.838237 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.837101    2580 factory.go:153] Registering CRI-O factory
May 29 01:58:23.838237 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.837111    2580 factory.go:223] Registration of the crio container factory successfully
May 29 01:58:23.838237 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.837124    2580 factory.go:103] Registering Raw factory
May 29 01:58:23.838237 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.837136    2580 manager.go:1196] Started watching for new ooms in manager
May 29 01:58:23.838237 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.837762    2580 manager.go:319] Starting recovery of all containers
May 29 01:58:23.848056 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.848024    2580 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 29 01:58:23.848416 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.848389    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 29 01:58:23.850275 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.850260    2580 manager.go:324] Recovery completed
May 29 01:58:23.854142 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.854129    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 01:58:23.855063 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.855044    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qnkqw"
May 29 01:58:23.857380 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.857367    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientMemory"
May 29 01:58:23.857426 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.857395    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasNoDiskPressure"
May 29 01:58:23.857426 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.857408    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientPID"
May 29 01:58:23.857917 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.857903    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 29 01:58:23.857917 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.857915    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 29 01:58:23.858032 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.857932    2580 state_mem.go:36] "Initialized new in-memory state store"
May 29 01:58:23.860750 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.860735    2580 policy_none.go:49] "None policy: Start"
May 29 01:58:23.860819 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.860753    2580 memory_manager.go:186] "Starting memorymanager" policy="None"
May 29 01:58:23.860819 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.860767    2580 state_mem.go:35] "Initializing new in-memory state store"
May 29 01:58:23.862399 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.862382    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qnkqw"
May 29 01:58:23.893259 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.893236    2580 manager.go:341] "Starting Device Plugin manager"
May 29 01:58:23.893354 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.893325    2580 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 29 01:58:23.893354 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.893349    2580 server.go:85] "Starting device plugin registration server"
May 29 01:58:23.893629 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.893616    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
May 29 01:58:23.893728 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.893631    2580 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 29 01:58:23.893838 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.893818    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 29 01:58:23.894040 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.893890    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 29 01:58:23.894040 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.893901    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 29 01:58:23.894521 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.894496    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 29 01:58:23.894785 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.894543    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:23.968355 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.968277    2580 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 29 01:58:23.969547 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.969531    2580 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 29 01:58:23.969634 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.969553    2580 status_manager.go:230] "Starting to sync pod status with apiserver"
May 29 01:58:23.969634 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.969574    2580 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 29 01:58:23.969634 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.969581    2580 kubelet.go:2451] "Starting kubelet main sync loop"
May 29 01:58:23.969785 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:23.969634    2580 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 29 01:58:23.973566 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.973549    2580 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 01:58:23.994476 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.994459    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 01:58:23.995669 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.995654    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientMemory"
May 29 01:58:23.995739 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.995681    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasNoDiskPressure"
May 29 01:58:23.995739 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.995692    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientPID"
May 29 01:58:23.995739 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:23.995715    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-62.ec2.internal"
May 29 01:58:24.009457 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.009434    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-62.ec2.internal"
May 29 01:58:24.009457 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.009459    2580 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-62.ec2.internal\": node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.060987 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.060964    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.069862 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.069828    2580 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal"]
May 29 01:58:24.069956 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.069902    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 01:58:24.071005 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.070988    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientMemory"
May 29 01:58:24.071089 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.071018    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasNoDiskPressure"
May 29 01:58:24.071089 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.071030    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientPID"
May 29 01:58:24.072479 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.072467    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 01:58:24.072670 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.072657    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.072712 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.072687    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 01:58:24.073328 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.073312    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientMemory"
May 29 01:58:24.073396 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.073319    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientMemory"
May 29 01:58:24.073396 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.073363    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasNoDiskPressure"
May 29 01:58:24.073396 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.073374    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientPID"
May 29 01:58:24.073487 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.073341    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasNoDiskPressure"
May 29 01:58:24.073487 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.073423    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientPID"
May 29 01:58:24.074817 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.074803    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.074888 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.074825    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 01:58:24.075709 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.075688    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientMemory"
May 29 01:58:24.075709 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.075713    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasNoDiskPressure"
May 29 01:58:24.075858 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.075724    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeHasSufficientPID"
May 29 01:58:24.100335 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.100313    2580 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-62.ec2.internal\" not found" node="ip-10-0-129-62.ec2.internal"
May 29 01:58:24.104787 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.104772    2580 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-62.ec2.internal\" not found" node="ip-10-0-129-62.ec2.internal"
May 29 01:58:24.137309 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.137287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3eb4714551012f862c3fbf2527f09f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal\" (UID: \"e3eb4714551012f862c3fbf2527f09f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.137410 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.137318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3eb4714551012f862c3fbf2527f09f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal\" (UID: \"e3eb4714551012f862c3fbf2527f09f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.137410 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.137335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6bfc788d18bccc31168708c81f074e78-config\") pod \"kube-apiserver-proxy-ip-10-0-129-62.ec2.internal\" (UID: \"6bfc788d18bccc31168708c81f074e78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.161635 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.161614    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.237628 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.237550    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3eb4714551012f862c3fbf2527f09f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal\" (UID: \"e3eb4714551012f862c3fbf2527f09f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.237628 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.237584    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3eb4714551012f862c3fbf2527f09f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal\" (UID: \"e3eb4714551012f862c3fbf2527f09f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.237628 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.237612    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6bfc788d18bccc31168708c81f074e78-config\") pod \"kube-apiserver-proxy-ip-10-0-129-62.ec2.internal\" (UID: \"6bfc788d18bccc31168708c81f074e78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.237769 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.237637    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6bfc788d18bccc31168708c81f074e78-config\") pod \"kube-apiserver-proxy-ip-10-0-129-62.ec2.internal\" (UID: \"6bfc788d18bccc31168708c81f074e78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.237769 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.237643    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3eb4714551012f862c3fbf2527f09f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal\" (UID: \"e3eb4714551012f862c3fbf2527f09f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.237769 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.237650    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3eb4714551012f862c3fbf2527f09f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal\" (UID: \"e3eb4714551012f862c3fbf2527f09f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.261886 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.261863    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.362345 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.362317    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.403488 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.403463    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.408255 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.408238    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal"
May 29 01:58:24.462715 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.462680    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.563232 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.563150    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.663678 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.663649    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.735954 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.735922    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 29 01:58:24.736613 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.736072    2580 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 29 01:58:24.764414 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.764387    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.835916 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.835889    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 29 01:58:24.839238 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.839219    2580 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 01:58:24.847316 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.847297    2580 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 29 01:58:24.864857 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.864820    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-28 01:53:23 +0000 UTC" deadline="2028-01-08 01:20:38.978135505 +0000 UTC"
May 29 01:58:24.864857 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.864843    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.864857 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.864851    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14135h22m14.11328795s"
May 29 01:58:24.893904 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.893872    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hpk8b"
May 29 01:58:24.903221 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.903204    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hpk8b"
May 29 01:58:24.951300 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:24.951268    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfc788d18bccc31168708c81f074e78.slice/crio-3cd17dcc94f879febcfa0c7965c41f04fa7eaf26f5da04bcb6d221defedbfa7e WatchSource:0}: Error finding container 3cd17dcc94f879febcfa0c7965c41f04fa7eaf26f5da04bcb6d221defedbfa7e: Status 404 returned error can't find the container with id 3cd17dcc94f879febcfa0c7965c41f04fa7eaf26f5da04bcb6d221defedbfa7e
May 29 01:58:24.951491 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:24.951468    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3eb4714551012f862c3fbf2527f09f1.slice/crio-685dd2f73908f0a6796617277efc5086bf68cb09338fa78e72656fb767e7d9b5 WatchSource:0}: Error finding container 685dd2f73908f0a6796617277efc5086bf68cb09338fa78e72656fb767e7d9b5: Status 404 returned error can't find the container with id 685dd2f73908f0a6796617277efc5086bf68cb09338fa78e72656fb767e7d9b5
May 29 01:58:24.956417 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.956403    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 01:58:24.965794 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:24.965771    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-62.ec2.internal\" not found"
May 29 01:58:24.972119 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.972061    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal" event={"ID":"6bfc788d18bccc31168708c81f074e78","Type":"ContainerStarted","Data":"3cd17dcc94f879febcfa0c7965c41f04fa7eaf26f5da04bcb6d221defedbfa7e"}
May 29 01:58:24.973000 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:24.972980    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal" event={"ID":"e3eb4714551012f862c3fbf2527f09f1","Type":"ContainerStarted","Data":"685dd2f73908f0a6796617277efc5086bf68cb09338fa78e72656fb767e7d9b5"}
May 29 01:58:25.020867 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.020833    2580 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 01:58:25.031538 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.031515    2580 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 01:58:25.036202 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.036186    2580 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal"
May 29 01:58:25.046732 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.046706    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 29 01:58:25.047831 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.047816    2580 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal"
May 29 01:58:25.078716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.078645    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 29 01:58:25.811017 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.810977    2580 apiserver.go:52] "Watching apiserver"
May 29 01:58:25.821205 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.821177    2580 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 29 01:58:25.823556 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.823525    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-zsdgm","kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57","openshift-cluster-node-tuning-operator/tuned-hj87r","openshift-image-registry/node-ca-lmdcx","openshift-network-operator/iptables-alerter-8trgn","openshift-ovn-kubernetes/ovnkube-node-pktj7","openshift-dns/node-resolver-kdkn5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal","openshift-multus/multus-additional-cni-plugins-9p47b","openshift-multus/multus-mt97d","openshift-multus/network-metrics-daemon-7475l","openshift-network-diagnostics/network-check-target-qfhm9"]
May 29 01:58:25.826089 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.826070    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.828163 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.828144    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.830482 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.830456    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.830572 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.830481    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 29 01:58:25.830902 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.830882    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 29 01:58:25.830999 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.830964    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 29 01:58:25.831130 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.831114    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 29 01:58:25.831532 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.831516    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 29 01:58:25.831637 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.831621    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 29 01:58:25.831804 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.831791    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 29 01:58:25.831856 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.831835    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cfc2q\""
May 29 01:58:25.832049 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.832033    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zkph2\""
May 29 01:58:25.832094 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.832039    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 29 01:58:25.832262 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.832245    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 29 01:58:25.832632 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.832609    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.834445 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.834401    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j6f2b\""
May 29 01:58:25.834544 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.834529    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 29 01:58:25.834863 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.834842    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.835581 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.835566    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 29 01:58:25.835975 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.835957    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rksbl\""
May 29 01:58:25.836071 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.836008    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 29 01:58:25.836129 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.835963    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 29 01:58:25.836294 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.836273    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 29 01:58:25.837665 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.837645    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:25.840108 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.840045    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:25.840569 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.840537    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 29 01:58:25.841238 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.841219    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 29 01:58:25.841345 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.841289    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4r2fk\""
May 29 01:58:25.841345 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.841331    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 29 01:58:25.842466 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.842153    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 29 01:58:25.842466 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.842339    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 29 01:58:25.842872 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.842856    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:25.843420 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.843403    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mt659\""
May 29 01:58:25.843940 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.843926    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7r9mf\""
May 29 01:58:25.844827 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.844802    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 29 01:58:25.844919 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.844831    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 29 01:58:25.845120 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.845104    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mt97d"
May 29 01:58:25.846677 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-sysctl-d\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.846766 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-systemd\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.846766 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58f43f17-039c-4be9-9db1-d1fb13349a95-konnectivity-ca\") pod \"konnectivity-agent-zsdgm\" (UID: \"58f43f17-039c-4be9-9db1-d1fb13349a95\") " pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:25.846766 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-sysconfig\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.846920 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-run\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.846920 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-sys\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.846920 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-lib-modules\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.846920 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a44c991-a799-4094-8818-44f1617948e4-iptables-alerter-script\") pod \"iptables-alerter-8trgn\" (UID: \"8a44c991-a799-4094-8818-44f1617948e4\") " pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.846920 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-run-netns\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.846920 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-node-log\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.846920 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-log-socket\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.846920 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e064a153-5596-4b6e-9de1-e7734036edc9-env-overrides\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2c5\" (UniqueName: \"kubernetes.io/projected/e064a153-5596-4b6e-9de1-e7734036edc9-kube-api-access-vm2c5\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846978    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.846966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-socket-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847131    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-device-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-systemd-units\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-run-systemd\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-var-lib-openvswitch\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e064a153-5596-4b6e-9de1-e7734036edc9-ovn-node-metrics-cert\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.847295 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlj7r\" (UniqueName: \"kubernetes.io/projected/b85086f6-c65d-4db3-ba11-63438c9c34e4-kube-api-access-hlj7r\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847314    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-sysctl-conf\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847342    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmxj\" (UniqueName: \"kubernetes.io/projected/8a44c991-a799-4094-8818-44f1617948e4-kube-api-access-msmxj\") pod \"iptables-alerter-8trgn\" (UID: \"8a44c991-a799-4094-8818-44f1617948e4\") " pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-registration-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-var-lib-kubelet\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-host\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-tuned\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847458    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lqk\" (UniqueName: \"kubernetes.io/projected/79261d39-6534-47f2-9d07-2e62121b5e6d-kube-api-access-s9lqk\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a44c991-a799-4094-8818-44f1617948e4-host-slash\") pod \"iptables-alerter-8trgn\" (UID: \"8a44c991-a799-4094-8818-44f1617948e4\") " pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-run-openvswitch\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e064a153-5596-4b6e-9de1-e7734036edc9-ovnkube-config\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-modprobe-d\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhph2\" (UniqueName: \"kubernetes.io/projected/41a9f0b2-1f52-4fb2-b920-617ce22dc8d0-kube-api-access-zhph2\") pod \"node-ca-lmdcx\" (UID: \"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0\") " pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.847716 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847513    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58f43f17-039c-4be9-9db1-d1fb13349a95-agent-certs\") pod \"konnectivity-agent-zsdgm\" (UID: \"58f43f17-039c-4be9-9db1-d1fb13349a95\") " pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847766    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ggfxm\""
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-cni-bin\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41a9f0b2-1f52-4fb2-b920-617ce22dc8d0-host\") pod \"node-ca-lmdcx\" (UID: \"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0\") " pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41a9f0b2-1f52-4fb2-b920-617ce22dc8d0-serviceca\") pod \"node-ca-lmdcx\" (UID: \"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0\") " pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847902    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rlb5k\""
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-slash\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.847970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-etc-openvswitch\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.848000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-run-ovn\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.848027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-kubernetes\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.848060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79261d39-6534-47f2-9d07-2e62121b5e6d-tmp\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.848282 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.848244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-kubelet\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.848941 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.848290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-cni-netd\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.848941 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.848331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e064a153-5596-4b6e-9de1-e7734036edc9-ovnkube-script-lib\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.848941 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.848375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-sys-fs\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.849906 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.849885    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:25.850011 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.849935    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:25.850069 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:25.850013    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:25.850125 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:25.850096    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:25.904710 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.904674    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 01:53:24 +0000 UTC" deadline="2028-03-09 00:41:39.136457838 +0000 UTC"
May 29 01:58:25.904710 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.904709    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15598h43m13.231752711s"
May 29 01:58:25.937915 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.937887    2580 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 29 01:58:25.948979 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.948948    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-etc-openvswitch\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.949159 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.948986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-cnibin\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.949159 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949019    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-kubelet\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.949706 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949653    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e064a153-5596-4b6e-9de1-e7734036edc9-ovnkube-script-lib\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.949831 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949718    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-sys-fs\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.949831 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949748    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-run\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.949831 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949052    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-etc-openvswitch\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.949831 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949777    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-sys\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.949831 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949786    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-kubelet\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.949831 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06088ba1-03c3-493f-9665-13dfd7b77b04-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:25.950096 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949859    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-sysconfig\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.950096 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae3f09b5-807b-46e2-a6ea-cb7d47ae16db-hosts-file\") pod \"node-resolver-kdkn5\" (UID: \"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db\") " pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:25.950096 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949908    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-sys-fs\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.950096 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-socket-dir-parent\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.950096 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-run-netns\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.950096 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.949972    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-run\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.950096 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-var-lib-cni-multus\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.950096 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950040    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-sys\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.950096 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-hostroot\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.950495 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950129    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a44c991-a799-4094-8818-44f1617948e4-iptables-alerter-script\") pod \"iptables-alerter-8trgn\" (UID: \"8a44c991-a799-4094-8818-44f1617948e4\") " pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.950495 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950162    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-node-log\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.950495 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950200    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2c5\" (UniqueName: \"kubernetes.io/projected/e064a153-5596-4b6e-9de1-e7734036edc9-kube-api-access-vm2c5\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.950495 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950357    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-node-log\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.950495 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950357    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-sysconfig\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.950495 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950283    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-device-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.950829 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950562    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-device-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.950829 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950696    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e064a153-5596-4b6e-9de1-e7734036edc9-ovnkube-script-lib\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.950829 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950790    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-sysctl-conf\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.950965 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-var-lib-cni-bin\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.950965 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950869    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.950965 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950926    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-var-lib-openvswitch\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.950965 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950959    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e064a153-5596-4b6e-9de1-e7734036edc9-ovn-node-metrics-cert\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.951136 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950970    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.951136 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950990    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.951136 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.950984    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-sysctl-conf\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.951136 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-os-release\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:25.951136 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-os-release\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.951136 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951079    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a44c991-a799-4094-8818-44f1617948e4-iptables-alerter-script\") pod \"iptables-alerter-8trgn\" (UID: \"8a44c991-a799-4094-8818-44f1617948e4\") " pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.951136 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951086    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.951136 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-run-k8s-cni-cncf-io\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.951471 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951031    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-var-lib-openvswitch\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.951471 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87dg\" (UniqueName: \"kubernetes.io/projected/2f9bfc3c-879e-4385-8c93-729bd0bac31d-kube-api-access-w87dg\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.951471 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951251    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-msmxj\" (UniqueName: \"kubernetes.io/projected/8a44c991-a799-4094-8818-44f1617948e4-kube-api-access-msmxj\") pod \"iptables-alerter-8trgn\" (UID: \"8a44c991-a799-4094-8818-44f1617948e4\") " pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.951471 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951459    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 29 01:58:25.951847 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951808    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-registration-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.951962 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-daemon-config\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.951962 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:25.952057 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.951978    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a44c991-a799-4094-8818-44f1617948e4-host-slash\") pod \"iptables-alerter-8trgn\" (UID: \"8a44c991-a799-4094-8818-44f1617948e4\") " pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.952057 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952008    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-run-openvswitch\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.952154 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-system-cni-dir\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:25.952154 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952101    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58f43f17-039c-4be9-9db1-d1fb13349a95-agent-certs\") pod \"konnectivity-agent-zsdgm\" (UID: \"58f43f17-039c-4be9-9db1-d1fb13349a95\") " pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:25.952154 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952130    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.952287 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f9bfc3c-879e-4385-8c93-729bd0bac31d-cni-binary-copy\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.952287 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:25.952287 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952271    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41a9f0b2-1f52-4fb2-b920-617ce22dc8d0-serviceca\") pod \"node-ca-lmdcx\" (UID: \"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0\") " pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.952413 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952310    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-slash\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.952413 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952337    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-run-ovn\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.952413 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952362    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-kubernetes\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.952413 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952390    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79261d39-6534-47f2-9d07-2e62121b5e6d-tmp\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.952609 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952421    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-cni-netd\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.952609 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952534    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-sysctl-d\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.952609 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952580    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-systemd\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.952739 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952625    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-lib-modules\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.952739 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nsls\" (UniqueName: \"kubernetes.io/projected/ae3f09b5-807b-46e2-a6ea-cb7d47ae16db-kube-api-access-8nsls\") pod \"node-resolver-kdkn5\" (UID: \"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db\") " pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:25.952739 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-system-cni-dir\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.952739 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952727    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58f43f17-039c-4be9-9db1-d1fb13349a95-konnectivity-ca\") pod \"konnectivity-agent-zsdgm\" (UID: \"58f43f17-039c-4be9-9db1-d1fb13349a95\") " pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:25.952907 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-cni-dir\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.952907 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952833    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-run-netns\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.952907 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952876    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-log-socket\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.953040 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952953    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e064a153-5596-4b6e-9de1-e7734036edc9-env-overrides\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.953040 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.952984    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-socket-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.953040 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:25.953174 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06088ba1-03c3-493f-9665-13dfd7b77b04-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:25.953174 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-run-multus-certs\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.953174 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953155    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-var-lib-kubelet\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.953301 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953209    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-systemd-units\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.953359 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953344    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-run-systemd\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.953654 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953399    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hlj7r\" (UniqueName: \"kubernetes.io/projected/b85086f6-c65d-4db3-ba11-63438c9c34e4-kube-api-access-hlj7r\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.953736 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06088ba1-03c3-493f-9665-13dfd7b77b04-cni-binary-copy\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:25.953736 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-etc-kubernetes\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.953843 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953779    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-host\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.953843 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953808    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-tuned\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.953843 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953837    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lqk\" (UniqueName: \"kubernetes.io/projected/79261d39-6534-47f2-9d07-2e62121b5e6d-kube-api-access-s9lqk\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.953968 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-conf-dir\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.953968 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953894    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e064a153-5596-4b6e-9de1-e7734036edc9-ovnkube-config\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.954053 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.953962    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-modprobe-d\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.954053 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4psb\" (UniqueName: \"kubernetes.io/projected/06088ba1-03c3-493f-9665-13dfd7b77b04-kube-api-access-j4psb\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:25.954142 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-var-lib-kubelet\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:25.954142 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954106    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zhph2\" (UniqueName: \"kubernetes.io/projected/41a9f0b2-1f52-4fb2-b920-617ce22dc8d0-kube-api-access-zhph2\") pod \"node-ca-lmdcx\" (UID: \"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0\") " pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.954349 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954311    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-run-systemd\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.954436 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954396    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-cni-netd\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.954515 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954462    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-lib-modules\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.954619 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954602    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-sysctl-d\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.954718 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954695    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a44c991-a799-4094-8818-44f1617948e4-host-slash\") pod \"iptables-alerter-8trgn\" (UID: \"8a44c991-a799-4094-8818-44f1617948e4\") " pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.954804 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954667    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58f43f17-039c-4be9-9db1-d1fb13349a95-konnectivity-ca\") pod \"konnectivity-agent-zsdgm\" (UID: \"58f43f17-039c-4be9-9db1-d1fb13349a95\") " pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:25.954804 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954748    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e064a153-5596-4b6e-9de1-e7734036edc9-ovn-node-metrics-cert\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.954804 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954748    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41a9f0b2-1f52-4fb2-b920-617ce22dc8d0-serviceca\") pod \"node-ca-lmdcx\" (UID: \"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0\") " pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.954804 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954805    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-systemd\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.955044 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954860    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-cni-bin\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.955044 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.954324    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-registration-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.955238 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955214    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.955312 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955240    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-cni-bin\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.955312 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955274    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-slash\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.955406 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955372    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b85086f6-c65d-4db3-ba11-63438c9c34e4-socket-dir\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.955406 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955378    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-run-ovn\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.955406 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955391    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-run-netns\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.955533 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955405    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-var-lib-kubelet\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.955533 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955400    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-kubernetes\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.955533 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955452    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-log-socket\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.955533 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955524    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-modprobe-d\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.955730 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955539    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-systemd-units\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.955730 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955580    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-run-openvswitch\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.955730 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955656    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79261d39-6534-47f2-9d07-2e62121b5e6d-host\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.955897 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3f09b5-807b-46e2-a6ea-cb7d47ae16db-tmp-dir\") pod \"node-resolver-kdkn5\" (UID: \"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db\") " pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:25.955957 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.955928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-cnibin\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:25.956094 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.956064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mcl5\" (UniqueName: \"kubernetes.io/projected/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-kube-api-access-8mcl5\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:25.956173 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.956153    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.956228 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.956177    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e064a153-5596-4b6e-9de1-e7734036edc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.956228 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.956198    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41a9f0b2-1f52-4fb2-b920-617ce22dc8d0-host\") pod \"node-ca-lmdcx\" (UID: \"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0\") " pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.956228 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.956155    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e064a153-5596-4b6e-9de1-e7734036edc9-env-overrides\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.956404 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.956364    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41a9f0b2-1f52-4fb2-b920-617ce22dc8d0-host\") pod \"node-ca-lmdcx\" (UID: \"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0\") " pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.957386 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.957242    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e064a153-5596-4b6e-9de1-e7734036edc9-ovnkube-config\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.958166 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.958148    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79261d39-6534-47f2-9d07-2e62121b5e6d-etc-tuned\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.958465 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.958429    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79261d39-6534-47f2-9d07-2e62121b5e6d-tmp\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:25.958613 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.958578    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58f43f17-039c-4be9-9db1-d1fb13349a95-agent-certs\") pod \"konnectivity-agent-zsdgm\" (UID: \"58f43f17-039c-4be9-9db1-d1fb13349a95\") " pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:25.962133 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.962113    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2c5\" (UniqueName: \"kubernetes.io/projected/e064a153-5596-4b6e-9de1-e7734036edc9-kube-api-access-vm2c5\") pod \"ovnkube-node-pktj7\" (UID: \"e064a153-5596-4b6e-9de1-e7734036edc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:25.962604 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.962570    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmxj\" (UniqueName: \"kubernetes.io/projected/8a44c991-a799-4094-8818-44f1617948e4-kube-api-access-msmxj\") pod \"iptables-alerter-8trgn\" (UID: \"8a44c991-a799-4094-8818-44f1617948e4\") " pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:25.964696 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.964657    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhph2\" (UniqueName: \"kubernetes.io/projected/41a9f0b2-1f52-4fb2-b920-617ce22dc8d0-kube-api-access-zhph2\") pod \"node-ca-lmdcx\" (UID: \"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0\") " pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:25.965166 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.965149    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlj7r\" (UniqueName: \"kubernetes.io/projected/b85086f6-c65d-4db3-ba11-63438c9c34e4-kube-api-access-hlj7r\") pod \"aws-ebs-csi-driver-node-q7k57\" (UID: \"b85086f6-c65d-4db3-ba11-63438c9c34e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:25.965641 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:25.965616    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lqk\" (UniqueName: \"kubernetes.io/projected/79261d39-6534-47f2-9d07-2e62121b5e6d-kube-api-access-s9lqk\") pod \"tuned-hj87r\" (UID: \"79261d39-6534-47f2-9d07-2e62121b5e6d\") " pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:26.049142 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.049111    2580 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 01:58:26.057026 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057001    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-socket-dir-parent\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057026 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057032    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-run-netns\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057223 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057049    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-var-lib-cni-multus\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057223 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057099    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-hostroot\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057223 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057118    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-var-lib-cni-bin\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057223 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057133    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-socket-dir-parent\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057223 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057171    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-os-release\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.057223 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057179    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-hostroot\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057223 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057192    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-run-netns\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057223 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057208    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-var-lib-cni-bin\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057212    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-os-release\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057267    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-run-k8s-cni-cncf-io\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057284    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w87dg\" (UniqueName: \"kubernetes.io/projected/2f9bfc3c-879e-4385-8c93-729bd0bac31d-kube-api-access-w87dg\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057311    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-daemon-config\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057319    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-os-release\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057328    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-os-release\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057334    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057221    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-var-lib-cni-multus\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057382    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-run-k8s-cni-cncf-io\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057500    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-system-cni-dir\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.057564 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057536    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-system-cni-dir\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057611    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f9bfc3c-879e-4385-8c93-729bd0bac31d-cni-binary-copy\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057640    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057678    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8nsls\" (UniqueName: \"kubernetes.io/projected/ae3f09b5-807b-46e2-a6ea-cb7d47ae16db-kube-api-access-8nsls\") pod \"node-resolver-kdkn5\" (UID: \"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db\") " pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057702    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-system-cni-dir\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057745    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-cni-dir\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.057781    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057786    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057813    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06088ba1-03c3-493f-9665-13dfd7b77b04-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.057880    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs podName:7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:26.557827 +0000 UTC m=+3.140817899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs") pod "network-metrics-daemon-7475l" (UID: "7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057904    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-daemon-config\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057954    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-run-multus-certs\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.057906    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-run-multus-certs\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058003    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06088ba1-03c3-493f-9665-13dfd7b77b04-cni-binary-copy\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058036    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-etc-kubernetes\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058060    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-conf-dir\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058084    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j4psb\" (UniqueName: \"kubernetes.io/projected/06088ba1-03c3-493f-9665-13dfd7b77b04-kube-api-access-j4psb\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058102    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-var-lib-kubelet\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058126    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3f09b5-807b-46e2-a6ea-cb7d47ae16db-tmp-dir\") pod \"node-resolver-kdkn5\" (UID: \"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db\") " pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058126    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-etc-kubernetes\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058144    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-cnibin\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058162    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f9bfc3c-879e-4385-8c93-729bd0bac31d-cni-binary-copy\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058175    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-conf-dir\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058173    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8mcl5\" (UniqueName: \"kubernetes.io/projected/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-kube-api-access-8mcl5\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058224    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-system-cni-dir\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058238    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-cnibin\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058275    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-host-var-lib-kubelet\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058283    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06088ba1-03c3-493f-9665-13dfd7b77b04-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058312    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae3f09b5-807b-46e2-a6ea-cb7d47ae16db-hosts-file\") pod \"node-resolver-kdkn5\" (UID: \"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db\") " pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058361    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058376    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06088ba1-03c3-493f-9665-13dfd7b77b04-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058383    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae3f09b5-807b-46e2-a6ea-cb7d47ae16db-hosts-file\") pod \"node-resolver-kdkn5\" (UID: \"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db\") " pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058248    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-multus-cni-dir\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058411    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f9bfc3c-879e-4385-8c93-729bd0bac31d-cnibin\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.058791 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058456    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3f09b5-807b-46e2-a6ea-cb7d47ae16db-tmp-dir\") pod \"node-resolver-kdkn5\" (UID: \"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db\") " pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:26.059479 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058508    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06088ba1-03c3-493f-9665-13dfd7b77b04-cnibin\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.059479 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058661    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06088ba1-03c3-493f-9665-13dfd7b77b04-cni-binary-copy\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.059479 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.058823    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06088ba1-03c3-493f-9665-13dfd7b77b04-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.071012 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.070947    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 01:58:26.071012 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.070972    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 01:58:26.071012 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.070986    2580 projected.go:194] Error preparing data for projected volume kube-api-access-mt49b for pod openshift-network-diagnostics/network-check-target-qfhm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:26.071216 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.071073    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b podName:734ad269-957d-44e9-9400-4270c2f494ae nodeName:}" failed. No retries permitted until 2026-05-29 01:58:26.57105485 +0000 UTC m=+3.154045760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mt49b" (UniqueName: "kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b") pod "network-check-target-qfhm9" (UID: "734ad269-957d-44e9-9400-4270c2f494ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:26.073819 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.073793    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4psb\" (UniqueName: \"kubernetes.io/projected/06088ba1-03c3-493f-9665-13dfd7b77b04-kube-api-access-j4psb\") pod \"multus-additional-cni-plugins-9p47b\" (UID: \"06088ba1-03c3-493f-9665-13dfd7b77b04\") " pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.081414 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.081394    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nsls\" (UniqueName: \"kubernetes.io/projected/ae3f09b5-807b-46e2-a6ea-cb7d47ae16db-kube-api-access-8nsls\") pod \"node-resolver-kdkn5\" (UID: \"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db\") " pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:26.088385 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.088357    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87dg\" (UniqueName: \"kubernetes.io/projected/2f9bfc3c-879e-4385-8c93-729bd0bac31d-kube-api-access-w87dg\") pod \"multus-mt97d\" (UID: \"2f9bfc3c-879e-4385-8c93-729bd0bac31d\") " pod="openshift-multus/multus-mt97d"
May 29 01:58:26.089275 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.089258    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mcl5\" (UniqueName: \"kubernetes.io/projected/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-kube-api-access-8mcl5\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:26.137938 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.137907    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:26.146439 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.146413    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57"
May 29 01:58:26.154838 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.154819    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hj87r"
May 29 01:58:26.159539 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.159521    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lmdcx"
May 29 01:58:26.167080 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.167060    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8trgn"
May 29 01:58:26.173675 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.173659    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:26.181460 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.181439    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kdkn5"
May 29 01:58:26.187370 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.187344    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9p47b"
May 29 01:58:26.191916 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.191899    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mt97d"
May 29 01:58:26.543853 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:26.543748    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9bfc3c_879e_4385_8c93_729bd0bac31d.slice/crio-5fa67cea01590ef43d1be75738af87c26243024bdce9b2d51981c8ee138b0029 WatchSource:0}: Error finding container 5fa67cea01590ef43d1be75738af87c26243024bdce9b2d51981c8ee138b0029: Status 404 returned error can't find the container with id 5fa67cea01590ef43d1be75738af87c26243024bdce9b2d51981c8ee138b0029
May 29 01:58:26.544709 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:26.544686    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79261d39_6534_47f2_9d07_2e62121b5e6d.slice/crio-d8a06ed8322cb04290e0f9e545ae8cc9414a5a4aec9d79a912a6bd47e9e6c477 WatchSource:0}: Error finding container d8a06ed8322cb04290e0f9e545ae8cc9414a5a4aec9d79a912a6bd47e9e6c477: Status 404 returned error can't find the container with id d8a06ed8322cb04290e0f9e545ae8cc9414a5a4aec9d79a912a6bd47e9e6c477
May 29 01:58:26.545664 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:26.545642    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a44c991_a799_4094_8818_44f1617948e4.slice/crio-dffa9ce4e2e1521239a7594f1b63628d45c90ac9ab2be8fadf9ca147509a3a54 WatchSource:0}: Error finding container dffa9ce4e2e1521239a7594f1b63628d45c90ac9ab2be8fadf9ca147509a3a54: Status 404 returned error can't find the container with id dffa9ce4e2e1521239a7594f1b63628d45c90ac9ab2be8fadf9ca147509a3a54
May 29 01:58:26.547011 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:26.546911    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06088ba1_03c3_493f_9665_13dfd7b77b04.slice/crio-d1a23c645de6c89d86330f2bf5dc8c61e52ee1d5d3fc7fc1a29be42d280af19f WatchSource:0}: Error finding container d1a23c645de6c89d86330f2bf5dc8c61e52ee1d5d3fc7fc1a29be42d280af19f: Status 404 returned error can't find the container with id d1a23c645de6c89d86330f2bf5dc8c61e52ee1d5d3fc7fc1a29be42d280af19f
May 29 01:58:26.549708 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:26.549688    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f43f17_039c_4be9_9db1_d1fb13349a95.slice/crio-be5a095ac52aab3c239048f07d8bb93906c2d300dfeaad4a089d99d5dd1a8dfc WatchSource:0}: Error finding container be5a095ac52aab3c239048f07d8bb93906c2d300dfeaad4a089d99d5dd1a8dfc: Status 404 returned error can't find the container with id be5a095ac52aab3c239048f07d8bb93906c2d300dfeaad4a089d99d5dd1a8dfc
May 29 01:58:26.550613 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:26.550563    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85086f6_c65d_4db3_ba11_63438c9c34e4.slice/crio-759c740d2ac27f08ae5ebcc17b3b59c9cf3a9ec757cdd53ca871baf36f96567f WatchSource:0}: Error finding container 759c740d2ac27f08ae5ebcc17b3b59c9cf3a9ec757cdd53ca871baf36f96567f: Status 404 returned error can't find the container with id 759c740d2ac27f08ae5ebcc17b3b59c9cf3a9ec757cdd53ca871baf36f96567f
May 29 01:58:26.551513 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:26.551459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3f09b5_807b_46e2_a6ea_cb7d47ae16db.slice/crio-c93138212959413d6d7e2e3439747a9a14257178e281e027e423494bbc6cb5e0 WatchSource:0}: Error finding container c93138212959413d6d7e2e3439747a9a14257178e281e027e423494bbc6cb5e0: Status 404 returned error can't find the container with id c93138212959413d6d7e2e3439747a9a14257178e281e027e423494bbc6cb5e0
May 29 01:58:26.553627 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:26.553511    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a9f0b2_1f52_4fb2_b920_617ce22dc8d0.slice/crio-ac3e586a14c0205fef55e54664d32cba82819105599da2df3e918f166a60d1b3 WatchSource:0}: Error finding container ac3e586a14c0205fef55e54664d32cba82819105599da2df3e918f166a60d1b3: Status 404 returned error can't find the container with id ac3e586a14c0205fef55e54664d32cba82819105599da2df3e918f166a60d1b3
May 29 01:58:26.553869 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:58:26.553845    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode064a153_5596_4b6e_9de1_e7734036edc9.slice/crio-9497b630e3d9f37c5837ec10968b0c7f066d6a45c8d3a8b17ade99eca87bb8fa WatchSource:0}: Error finding container 9497b630e3d9f37c5837ec10968b0c7f066d6a45c8d3a8b17ade99eca87bb8fa: Status 404 returned error can't find the container with id 9497b630e3d9f37c5837ec10968b0c7f066d6a45c8d3a8b17ade99eca87bb8fa
May 29 01:58:26.561417 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.561394    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:26.561624 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.561553    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:26.561671 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.561627    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs podName:7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:27.561606862 +0000 UTC m=+4.144597765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs") pod "network-metrics-daemon-7475l" (UID: "7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:26.661866 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.661830    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:26.662045 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.661987    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 01:58:26.662045 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.662007    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 01:58:26.662045 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.662017    2580 projected.go:194] Error preparing data for projected volume kube-api-access-mt49b for pod openshift-network-diagnostics/network-check-target-qfhm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:26.662139 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.662074    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b podName:734ad269-957d-44e9-9400-4270c2f494ae nodeName:}" failed. No retries permitted until 2026-05-29 01:58:27.662060872 +0000 UTC m=+4.245051755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt49b" (UniqueName: "kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b") pod "network-check-target-qfhm9" (UID: "734ad269-957d-44e9-9400-4270c2f494ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:26.905301 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.905208    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 01:53:24 +0000 UTC" deadline="2027-12-07 11:40:29.617989462 +0000 UTC"
May 29 01:58:26.905301 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.905246    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13377h42m2.712747457s"
May 29 01:58:26.970545 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.970516    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:26.970732 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:26.970698    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:26.984224 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.984106    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerStarted","Data":"9497b630e3d9f37c5837ec10968b0c7f066d6a45c8d3a8b17ade99eca87bb8fa"}
May 29 01:58:26.986750 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.986690    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kdkn5" event={"ID":"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db","Type":"ContainerStarted","Data":"c93138212959413d6d7e2e3439747a9a14257178e281e027e423494bbc6cb5e0"}
May 29 01:58:26.990677 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.990618    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p47b" event={"ID":"06088ba1-03c3-493f-9665-13dfd7b77b04","Type":"ContainerStarted","Data":"d1a23c645de6c89d86330f2bf5dc8c61e52ee1d5d3fc7fc1a29be42d280af19f"}
May 29 01:58:26.995069 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.994750    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8trgn" event={"ID":"8a44c991-a799-4094-8818-44f1617948e4","Type":"ContainerStarted","Data":"dffa9ce4e2e1521239a7594f1b63628d45c90ac9ab2be8fadf9ca147509a3a54"}
May 29 01:58:26.998750 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:26.998725    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal" event={"ID":"6bfc788d18bccc31168708c81f074e78","Type":"ContainerStarted","Data":"ecd3a7d6ebd1d00dc35cf12e96486549eeb2d313eb624b9298572b690df8bbc8"}
May 29 01:58:27.000156 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.000114    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57" event={"ID":"b85086f6-c65d-4db3-ba11-63438c9c34e4","Type":"ContainerStarted","Data":"759c740d2ac27f08ae5ebcc17b3b59c9cf3a9ec757cdd53ca871baf36f96567f"}
May 29 01:58:27.003396 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.003328    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lmdcx" event={"ID":"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0","Type":"ContainerStarted","Data":"ac3e586a14c0205fef55e54664d32cba82819105599da2df3e918f166a60d1b3"}
May 29 01:58:27.005374 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.005324    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zsdgm" event={"ID":"58f43f17-039c-4be9-9db1-d1fb13349a95","Type":"ContainerStarted","Data":"be5a095ac52aab3c239048f07d8bb93906c2d300dfeaad4a089d99d5dd1a8dfc"}
May 29 01:58:27.006571 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.006530    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hj87r" event={"ID":"79261d39-6534-47f2-9d07-2e62121b5e6d","Type":"ContainerStarted","Data":"d8a06ed8322cb04290e0f9e545ae8cc9414a5a4aec9d79a912a6bd47e9e6c477"}
May 29 01:58:27.008398 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.008328    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mt97d" event={"ID":"2f9bfc3c-879e-4385-8c93-729bd0bac31d","Type":"ContainerStarted","Data":"5fa67cea01590ef43d1be75738af87c26243024bdce9b2d51981c8ee138b0029"}
May 29 01:58:27.012540 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.011975    2580 generic.go:358] "Generic (PLEG): container finished" podID="e3eb4714551012f862c3fbf2527f09f1" containerID="c8702204b795ddbf2701fb4929961559855ed9592b7c2dba5071d56bdc61fabb" exitCode=0
May 29 01:58:27.012540 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.012006    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal" event={"ID":"e3eb4714551012f862c3fbf2527f09f1","Type":"ContainerDied","Data":"c8702204b795ddbf2701fb4929961559855ed9592b7c2dba5071d56bdc61fabb"}
May 29 01:58:27.022464 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.021313    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-62.ec2.internal" podStartSLOduration=2.02130003 podStartE2EDuration="2.02130003s" podCreationTimestamp="2026-05-29 01:58:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 01:58:27.019466889 +0000 UTC m=+3.602457793" watchObservedRunningTime="2026-05-29 01:58:27.02130003 +0000 UTC m=+3.604290936"
May 29 01:58:27.570633 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.570532    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:27.570790 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:27.570689    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:27.570790 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:27.570751    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs podName:7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:29.570732411 +0000 UTC m=+6.153723310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs") pod "network-metrics-daemon-7475l" (UID: "7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:27.671686 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.671086    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:27.671686 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:27.671244    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 01:58:27.671686 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:27.671265    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 01:58:27.671686 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:27.671277    2580 projected.go:194] Error preparing data for projected volume kube-api-access-mt49b for pod openshift-network-diagnostics/network-check-target-qfhm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:27.671686 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:27.671332    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b podName:734ad269-957d-44e9-9400-4270c2f494ae nodeName:}" failed. No retries permitted until 2026-05-29 01:58:29.671315038 +0000 UTC m=+6.254305920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt49b" (UniqueName: "kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b") pod "network-check-target-qfhm9" (UID: "734ad269-957d-44e9-9400-4270c2f494ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:27.972665 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:27.972570    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:27.973195 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:27.972731    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:28.026172 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:28.026138    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal" event={"ID":"e3eb4714551012f862c3fbf2527f09f1","Type":"ContainerStarted","Data":"845ba5f2201e97de5a29739c73755305a33d697cc0d4afa84d733f2b1ffe0c3d"}
May 29 01:58:28.053602 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:28.053183    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-62.ec2.internal" podStartSLOduration=3.053164478 podStartE2EDuration="3.053164478s" podCreationTimestamp="2026-05-29 01:58:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 01:58:28.052835739 +0000 UTC m=+4.635826646" watchObservedRunningTime="2026-05-29 01:58:28.053164478 +0000 UTC m=+4.636155381"
May 29 01:58:28.971177 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:28.970619    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:28.971177 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:28.970747    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:29.586314 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:29.586106    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:29.586314 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:29.586295    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:29.586830 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:29.586365    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs podName:7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:33.586346934 +0000 UTC m=+10.169337820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs") pod "network-metrics-daemon-7475l" (UID: "7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:29.686768 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:29.686704    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:29.686991 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:29.686969    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 01:58:29.687077 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:29.686999    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 01:58:29.687077 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:29.687012    2580 projected.go:194] Error preparing data for projected volume kube-api-access-mt49b for pod openshift-network-diagnostics/network-check-target-qfhm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:29.687175 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:29.687080    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b podName:734ad269-957d-44e9-9400-4270c2f494ae nodeName:}" failed. No retries permitted until 2026-05-29 01:58:33.687059285 +0000 UTC m=+10.270050183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt49b" (UniqueName: "kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b") pod "network-check-target-qfhm9" (UID: "734ad269-957d-44e9-9400-4270c2f494ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:29.970602 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:29.970424    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:29.970760 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:29.970587    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:30.971092 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:30.970703    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:30.971092 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:30.970829    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:31.970861 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:31.970826    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:31.971026 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:31.970974    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:32.970496 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:32.970453    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:32.970927 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:32.970554    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:33.617063 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:33.617022    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:33.617231 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:33.617198    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:33.617287 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:33.617261    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs podName:7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:41.617242305 +0000 UTC m=+18.200233191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs") pod "network-metrics-daemon-7475l" (UID: "7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:33.717997 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:33.717910    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:33.718178 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:33.718079    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 01:58:33.718178 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:33.718105    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 01:58:33.718178 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:33.718120    2580 projected.go:194] Error preparing data for projected volume kube-api-access-mt49b for pod openshift-network-diagnostics/network-check-target-qfhm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:33.718178 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:33.718179    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b podName:734ad269-957d-44e9-9400-4270c2f494ae nodeName:}" failed. No retries permitted until 2026-05-29 01:58:41.71815918 +0000 UTC m=+18.301150082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt49b" (UniqueName: "kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b") pod "network-check-target-qfhm9" (UID: "734ad269-957d-44e9-9400-4270c2f494ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:33.971204 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:33.970630    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:33.971204 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:33.970774    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:34.970622 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:34.970571    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:34.970788 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:34.970725    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:35.970677 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:35.970649    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:35.971133 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:35.970765    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:36.970793 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:36.970754    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:36.971161 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:36.970895    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:37.970334 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:37.970299    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:37.970496 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:37.970415    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:38.970754 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:38.970713    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:38.971208 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:38.970848    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:39.972891 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:39.972860    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:39.973308 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:39.972978    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:40.970149 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:40.970108    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:40.970331 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:40.970255    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:41.676085 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:41.676047    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:41.676572 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:41.676189    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:41.676572 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:41.676251    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs podName:7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:57.676233333 +0000 UTC m=+34.259224221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs") pod "network-metrics-daemon-7475l" (UID: "7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:41.776729 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:41.776692    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:41.776887 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:41.776850    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 01:58:41.776887 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:41.776871    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 01:58:41.776887 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:41.776880    2580 projected.go:194] Error preparing data for projected volume kube-api-access-mt49b for pod openshift-network-diagnostics/network-check-target-qfhm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:41.777024 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:41.776928    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b podName:734ad269-957d-44e9-9400-4270c2f494ae nodeName:}" failed. No retries permitted until 2026-05-29 01:58:57.776915005 +0000 UTC m=+34.359905894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt49b" (UniqueName: "kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b") pod "network-check-target-qfhm9" (UID: "734ad269-957d-44e9-9400-4270c2f494ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:41.970570 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:41.970488    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:41.970751 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:41.970637    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:42.970414 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:42.970384    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:42.970870 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:42.970493    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:43.971435 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:43.971254    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:43.971901 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:43.971514    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:44.971094 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:44.970932    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:44.971202 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:44.971174    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:45.059700 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.059675    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 01:58:45.060484 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.060189    2580 generic.go:358] "Generic (PLEG): container finished" podID="e064a153-5596-4b6e-9de1-e7734036edc9" containerID="ce30a5cd23543f207f668d3c1fd32026e73bdc2a902864269e809bd6ea12e088" exitCode=1
May 29 01:58:45.060484 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.060263    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerStarted","Data":"9c48fd39ea3b3b628e39557213e91799d40aa5af812611bcf840c80d3e0236a3"}
May 29 01:58:45.060484 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.060303    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerStarted","Data":"aaa5afa7ba796cdfd1e81f79dc2838d8051b95bbddac78694b9504e00b0b7ec0"}
May 29 01:58:45.060484 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.060318    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerStarted","Data":"a6ccc1cf73c30971c9aa47862137d4a03478fe3a6e5025f26ac4a8f079090206"}
May 29 01:58:45.060484 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.060332    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerDied","Data":"ce30a5cd23543f207f668d3c1fd32026e73bdc2a902864269e809bd6ea12e088"}
May 29 01:58:45.060484 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.060348    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerStarted","Data":"3b9dce0d6a442bc803cb1a7a8c394d6b4fbd803ee014f4cd49e360c20113bc01"}
May 29 01:58:45.062128 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.062075    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kdkn5" event={"ID":"ae3f09b5-807b-46e2-a6ea-cb7d47ae16db","Type":"ContainerStarted","Data":"f9bf82c6c1e6097983de16988c7c7fc7c6b9a08f5ca2a1aa08f7618871d7978d"}
May 29 01:58:45.063874 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.063848    2580 generic.go:358] "Generic (PLEG): container finished" podID="06088ba1-03c3-493f-9665-13dfd7b77b04" containerID="fa0bd1c7f370de76afc95f0bb545e2eb0e8dc7c1594ef727bc3befe9fcf35c70" exitCode=0
May 29 01:58:45.063968 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.063922    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p47b" event={"ID":"06088ba1-03c3-493f-9665-13dfd7b77b04","Type":"ContainerDied","Data":"fa0bd1c7f370de76afc95f0bb545e2eb0e8dc7c1594ef727bc3befe9fcf35c70"}
May 29 01:58:45.065725 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.065697    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57" event={"ID":"b85086f6-c65d-4db3-ba11-63438c9c34e4","Type":"ContainerStarted","Data":"ae7be719de9976cc45bc123ccd4adcf4bf0a74a359d8a4bf832bc5567d5012a7"}
May 29 01:58:45.067379 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.067341    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lmdcx" event={"ID":"41a9f0b2-1f52-4fb2-b920-617ce22dc8d0","Type":"ContainerStarted","Data":"5d6e48b1ca4173abc550580ecd9a9220aa5584c3622bd0f9837590e7649b7bf6"}
May 29 01:58:45.068974 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.068951    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zsdgm" event={"ID":"58f43f17-039c-4be9-9db1-d1fb13349a95","Type":"ContainerStarted","Data":"71abc61e82e82e2963f11bd28bebb9af0a88b3d10d93e3e25ced80c083b9a70c"}
May 29 01:58:45.070761 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.070736    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hj87r" event={"ID":"79261d39-6534-47f2-9d07-2e62121b5e6d","Type":"ContainerStarted","Data":"e8d419ecc8f77551fe5253c681f9bcede7cfebd49fcfa0548a48d2cf5cca52c0"}
May 29 01:58:45.073254 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.073227    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mt97d" event={"ID":"2f9bfc3c-879e-4385-8c93-729bd0bac31d","Type":"ContainerStarted","Data":"7ccc075901fd09e8a0f6a7a67f031df0c83d1123a7adbb96d4bb6dc849b4bd7b"}
May 29 01:58:45.104563 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.101439    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-kdkn5" podStartSLOduration=3.241530517 podStartE2EDuration="21.101418848s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:58:26.553275333 +0000 UTC m=+3.136266219" lastFinishedPulling="2026-05-29 01:58:44.413163653 +0000 UTC m=+20.996154550" observedRunningTime="2026-05-29 01:58:45.078668319 +0000 UTC m=+21.661659226" watchObservedRunningTime="2026-05-29 01:58:45.101418848 +0000 UTC m=+21.684409754"
May 29 01:58:45.124411 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.124354    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zsdgm" podStartSLOduration=3.262637539 podStartE2EDuration="21.124335886s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:58:26.55146288 +0000 UTC m=+3.134453778" lastFinishedPulling="2026-05-29 01:58:44.413161237 +0000 UTC m=+20.996152125" observedRunningTime="2026-05-29 01:58:45.099779669 +0000 UTC m=+21.682770575" watchObservedRunningTime="2026-05-29 01:58:45.124335886 +0000 UTC m=+21.707326792"
May 29 01:58:45.158226 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.158174    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-mt97d" podStartSLOduration=3.265695767 podStartE2EDuration="21.158159173s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:58:26.545519782 +0000 UTC m=+3.128510668" lastFinishedPulling="2026-05-29 01:58:44.437983191 +0000 UTC m=+21.020974074" observedRunningTime="2026-05-29 01:58:45.143253518 +0000 UTC m=+21.726244423" watchObservedRunningTime="2026-05-29 01:58:45.158159173 +0000 UTC m=+21.741150077"
May 29 01:58:45.184853 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.184795    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hj87r" podStartSLOduration=3.3156962549999998 podStartE2EDuration="21.184776692s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:58:26.547193479 +0000 UTC m=+3.130184371" lastFinishedPulling="2026-05-29 01:58:44.416273911 +0000 UTC m=+20.999264808" observedRunningTime="2026-05-29 01:58:45.157845803 +0000 UTC m=+21.740836702" watchObservedRunningTime="2026-05-29 01:58:45.184776692 +0000 UTC m=+21.767767597"
May 29 01:58:45.185027 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.184941    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lmdcx" podStartSLOduration=11.331877579 podStartE2EDuration="21.184933306s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:58:26.555144218 +0000 UTC m=+3.138135100" lastFinishedPulling="2026-05-29 01:58:36.408199921 +0000 UTC m=+12.991190827" observedRunningTime="2026-05-29 01:58:45.183959392 +0000 UTC m=+21.766950314" watchObservedRunningTime="2026-05-29 01:58:45.184933306 +0000 UTC m=+21.767924212"
May 29 01:58:45.838694 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.838665    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 29 01:58:45.909069 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.908945    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-29T01:58:45.838688637Z","UUID":"dab7b1b8-c765-4ac5-9cfa-9cc20ea3b4d4","Handler":null,"Name":"","Endpoint":""}
May 29 01:58:45.911367 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.911336    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 29 01:58:45.911367 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.911368    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 29 01:58:45.974551 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:45.974474    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:45.974727 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:45.974625    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:46.038315 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:46.038286    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:46.038953 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:46.038928    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:46.076907 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:46.076874    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57" event={"ID":"b85086f6-c65d-4db3-ba11-63438c9c34e4","Type":"ContainerStarted","Data":"1f275b80f8dec84e8b459ab9b67e9f62ac2ed8209b5cc5cf1e3fd3afe584b6fc"}
May 29 01:58:46.079748 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:46.079730    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 01:58:46.080072 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:46.080051    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerStarted","Data":"f86e133200d7bc96c3f8377f339445a4901051e9e89e8eff10359242f0584971"}
May 29 01:58:46.081533 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:46.081447    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8trgn" event={"ID":"8a44c991-a799-4094-8818-44f1617948e4","Type":"ContainerStarted","Data":"99ee140c2087b4b5ff21ff2100abff71ae2fa5555ad7943a3c22b48005515559"}
May 29 01:58:46.099290 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:46.099241    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8trgn" podStartSLOduration=4.233117656 podStartE2EDuration="22.099227518s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:58:26.547430257 +0000 UTC m=+3.130421158" lastFinishedPulling="2026-05-29 01:58:44.413540123 +0000 UTC m=+20.996531020" observedRunningTime="2026-05-29 01:58:46.098836083 +0000 UTC m=+22.681826983" watchObservedRunningTime="2026-05-29 01:58:46.099227518 +0000 UTC m=+22.682218422"
May 29 01:58:46.970319 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:46.970296    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:46.970468 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:46.970390    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:47.086027 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:47.085993    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57" event={"ID":"b85086f6-c65d-4db3-ba11-63438c9c34e4","Type":"ContainerStarted","Data":"adf4fdb1daf3424294fffdbd68726ffdb5f85fd255bd00e57d84ca55fa7355dc"}
May 29 01:58:47.086490 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:47.086073    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 29 01:58:47.970528 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:47.970497    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:47.970700 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:47.970661    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:48.090926 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:48.090898    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 01:58:48.091437 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:48.091225    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerStarted","Data":"119c344fba1d608ed3f3402cb1443361cab9aba30b4f71544066c353ef38ad5e"}
May 29 01:58:48.970466 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:48.970437    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:48.970649 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:48.970548    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:49.970647 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:49.970388    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:49.971344 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:49.970664    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:50.096730 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.096694    2580 generic.go:358] "Generic (PLEG): container finished" podID="06088ba1-03c3-493f-9665-13dfd7b77b04" containerID="19a95a928198f328f6ebd12b7b5e979bec49eb5a66ee49174b6c3dc88d3fa8af" exitCode=0
May 29 01:58:50.096872 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.096777    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p47b" event={"ID":"06088ba1-03c3-493f-9665-13dfd7b77b04","Type":"ContainerDied","Data":"19a95a928198f328f6ebd12b7b5e979bec49eb5a66ee49174b6c3dc88d3fa8af"}
May 29 01:58:50.099860 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.099841    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 01:58:50.100219 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.100193    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerStarted","Data":"f1322bd9a6c920a6fec97f4fa5cfe4989f0b4a1d0c62779dd5a3acab14d05c4f"}
May 29 01:58:50.100484 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.100467    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:50.100550 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.100497    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:50.100746 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.100730    2580 scope.go:117] "RemoveContainer" containerID="ce30a5cd23543f207f668d3c1fd32026e73bdc2a902864269e809bd6ea12e088"
May 29 01:58:50.117250 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.117228    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:50.119612 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.119552    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q7k57" podStartSLOduration=5.704252504 podStartE2EDuration="26.119536093s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:58:26.552657112 +0000 UTC m=+3.135647999" lastFinishedPulling="2026-05-29 01:58:46.967940705 +0000 UTC m=+23.550931588" observedRunningTime="2026-05-29 01:58:47.105964481 +0000 UTC m=+23.688955387" watchObservedRunningTime="2026-05-29 01:58:50.119536093 +0000 UTC m=+26.702526993"
May 29 01:58:50.970147 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:50.970118    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:50.970280 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:50.970214    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:51.105013 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.104945    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 01:58:51.105389 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.105335    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" event={"ID":"e064a153-5596-4b6e-9de1-e7734036edc9","Type":"ContainerStarted","Data":"0824b3a71a58b3e7476b80a1d8a315a0fbc0fa2da279ead72d2498d730ec57c3"}
May 29 01:58:51.105794 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.105766    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:51.107429 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.107403    2580 generic.go:358] "Generic (PLEG): container finished" podID="06088ba1-03c3-493f-9665-13dfd7b77b04" containerID="e444248b14518a09af09f113f88fb5c208d399a1332bf2445387837a5b500661" exitCode=0
May 29 01:58:51.107531 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.107440    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p47b" event={"ID":"06088ba1-03c3-493f-9665-13dfd7b77b04","Type":"ContainerDied","Data":"e444248b14518a09af09f113f88fb5c208d399a1332bf2445387837a5b500661"}
May 29 01:58:51.123408 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.123375    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:58:51.172549 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.172481    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7" podStartSLOduration=9.262049293 podStartE2EDuration="27.172462314s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:58:26.555567644 +0000 UTC m=+3.138558540" lastFinishedPulling="2026-05-29 01:58:44.465980675 +0000 UTC m=+21.048971561" observedRunningTime="2026-05-29 01:58:51.142382834 +0000 UTC m=+27.725373750" watchObservedRunningTime="2026-05-29 01:58:51.172462314 +0000 UTC m=+27.755453221"
May 29 01:58:51.459245 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.458966    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qfhm9"]
May 29 01:58:51.459402 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.459308    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:51.459465 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:51.459416    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:51.459802 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.459773    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7475l"]
May 29 01:58:51.459921 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:51.459893    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:51.460019 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:51.459995    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:52.111190 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:52.111097    2580 generic.go:358] "Generic (PLEG): container finished" podID="06088ba1-03c3-493f-9665-13dfd7b77b04" containerID="5bb580a545c484fd65540d8861d39eaf544779002c8695796a805a1bf19cc44d" exitCode=0
May 29 01:58:52.111190 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:52.111177    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p47b" event={"ID":"06088ba1-03c3-493f-9665-13dfd7b77b04","Type":"ContainerDied","Data":"5bb580a545c484fd65540d8861d39eaf544779002c8695796a805a1bf19cc44d"}
May 29 01:58:52.970712 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:52.970679    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:52.970876 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:52.970691    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:52.970876 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:52.970784    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:52.971018 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:52.970871    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:53.291524 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:53.291437    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:53.291937 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:53.291580    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 29 01:58:53.292350 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:53.292327    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zsdgm"
May 29 01:58:54.969760 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:54.969733    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:54.970475 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:54.969743    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:54.970475 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:54.969868    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:54.970475 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:54.969928    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:56.970400 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:56.970359    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:56.970400 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:56.970382    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:56.971037 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:56.970505    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfhm9" podUID="734ad269-957d-44e9-9400-4270c2f494ae"
May 29 01:58:56.971037 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:56.970645    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7475l" podUID="7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9"
May 29 01:58:57.196956 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.196753    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-62.ec2.internal" event="NodeReady"
May 29 01:58:57.197124 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.197070    2580 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 29 01:58:57.291082 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.290923    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lbjht"]
May 29 01:58:57.335355 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.335320    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fqm6l"]
May 29 01:58:57.335517 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.335475    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:58:57.344393 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.344360    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 29 01:58:57.345151 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.345001    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 29 01:58:57.345151 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.345071    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 29 01:58:57.345326 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.345236    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fmgpb\""
May 29 01:58:57.348661 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.348641    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lbjht"]
May 29 01:58:57.348785 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.348669    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fqm6l"]
May 29 01:58:57.348785 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.348772    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.352681 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.352518    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 29 01:58:57.352681 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.352553    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 29 01:58:57.352981 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.352964    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7t46n\""
May 29 01:58:57.488112 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.488069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnmck\" (UniqueName: \"kubernetes.io/projected/2cd392d5-8760-45f1-9164-4d9c714a6de3-kube-api-access-tnmck\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.488112 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.488107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:58:57.488383 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.488127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd392d5-8760-45f1-9164-4d9c714a6de3-tmp-dir\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.488383 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.488191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sh9l\" (UniqueName: \"kubernetes.io/projected/d21fba59-d15e-4394-bf25-23ba74c209e4-kube-api-access-5sh9l\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:58:57.488383 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.488263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.488383 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.488303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd392d5-8760-45f1-9164-4d9c714a6de3-config-volume\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.588738 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.588702    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:58:57.588738 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.588742    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd392d5-8760-45f1-9164-4d9c714a6de3-tmp-dir\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.588988 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.588771    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5sh9l\" (UniqueName: \"kubernetes.io/projected/d21fba59-d15e-4394-bf25-23ba74c209e4-kube-api-access-5sh9l\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:58:57.588988 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.588819    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.588988 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.588863    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 01:58:57.588988 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.588867    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd392d5-8760-45f1-9164-4d9c714a6de3-config-volume\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.588988 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.588965    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert podName:d21fba59-d15e-4394-bf25-23ba74c209e4 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:58.088942788 +0000 UTC m=+34.671933673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert") pod "ingress-canary-lbjht" (UID: "d21fba59-d15e-4394-bf25-23ba74c209e4") : secret "canary-serving-cert" not found
May 29 01:58:57.588988 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.588964    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 01:58:57.589221 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.589030    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tnmck\" (UniqueName: \"kubernetes.io/projected/2cd392d5-8760-45f1-9164-4d9c714a6de3-kube-api-access-tnmck\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.589221 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.589042    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls podName:2cd392d5-8760-45f1-9164-4d9c714a6de3 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:58.089022841 +0000 UTC m=+34.672013743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls") pod "dns-default-fqm6l" (UID: "2cd392d5-8760-45f1-9164-4d9c714a6de3") : secret "dns-default-metrics-tls" not found
May 29 01:58:57.589221 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.589131    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd392d5-8760-45f1-9164-4d9c714a6de3-tmp-dir\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.589357 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.589341    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2cd392d5-8760-45f1-9164-4d9c714a6de3-config-volume\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.602159 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.602125    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnmck\" (UniqueName: \"kubernetes.io/projected/2cd392d5-8760-45f1-9164-4d9c714a6de3-kube-api-access-tnmck\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:57.602315 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.602181    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sh9l\" (UniqueName: \"kubernetes.io/projected/d21fba59-d15e-4394-bf25-23ba74c209e4-kube-api-access-5sh9l\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:58:57.689701 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.689671    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:57.689878 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.689804    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:57.689947 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.689890    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs podName:7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9 nodeName:}" failed. No retries permitted until 2026-05-29 01:59:29.689859381 +0000 UTC m=+66.272850279 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs") pod "network-metrics-daemon-7475l" (UID: "7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 01:58:57.790582 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:57.790560    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:57.790729 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.790715    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 01:58:57.790767 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.790734    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 01:58:57.790767 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.790744    2580 projected.go:194] Error preparing data for projected volume kube-api-access-mt49b for pod openshift-network-diagnostics/network-check-target-qfhm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:57.790828 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:57.790784    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b podName:734ad269-957d-44e9-9400-4270c2f494ae nodeName:}" failed. No retries permitted until 2026-05-29 01:59:29.790772094 +0000 UTC m=+66.373762977 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt49b" (UniqueName: "kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b") pod "network-check-target-qfhm9" (UID: "734ad269-957d-44e9-9400-4270c2f494ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 01:58:58.093018 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.092988    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:58:58.093310 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.093043    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:58.093310 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:58.093126    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 01:58:58.093310 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:58.093140    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 01:58:58.093310 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:58.093189    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls podName:2cd392d5-8760-45f1-9164-4d9c714a6de3 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:59.093175748 +0000 UTC m=+35.676166631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls") pod "dns-default-fqm6l" (UID: "2cd392d5-8760-45f1-9164-4d9c714a6de3") : secret "dns-default-metrics-tls" not found
May 29 01:58:58.093310 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:58.093201    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert podName:d21fba59-d15e-4394-bf25-23ba74c209e4 nodeName:}" failed. No retries permitted until 2026-05-29 01:58:59.093195229 +0000 UTC m=+35.676186112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert") pod "ingress-canary-lbjht" (UID: "d21fba59-d15e-4394-bf25-23ba74c209e4") : secret "canary-serving-cert" not found
May 29 01:58:58.126005 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.125942    2580 generic.go:358] "Generic (PLEG): container finished" podID="06088ba1-03c3-493f-9665-13dfd7b77b04" containerID="300050b290b78f362756f430ffc4d867911e70287399a7f71485f7f11405f0f5" exitCode=0
May 29 01:58:58.126005 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.125976    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p47b" event={"ID":"06088ba1-03c3-493f-9665-13dfd7b77b04","Type":"ContainerDied","Data":"300050b290b78f362756f430ffc4d867911e70287399a7f71485f7f11405f0f5"}
May 29 01:58:58.970484 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.970404    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:58:58.970484 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.970465    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:58:58.975471 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.975447    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 29 01:58:58.975627 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.975609    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27b64\""
May 29 01:58:58.976268 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.976248    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbkht\""
May 29 01:58:58.976268 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.976264    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 29 01:58:58.976427 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:58.976297    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 29 01:58:59.100759 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:59.100724    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:58:59.101162 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:59.100807    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:58:59.101162 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:59.100858    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 01:58:59.101162 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:59.100917    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls podName:2cd392d5-8760-45f1-9164-4d9c714a6de3 nodeName:}" failed. No retries permitted until 2026-05-29 01:59:01.100902135 +0000 UTC m=+37.683893021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls") pod "dns-default-fqm6l" (UID: "2cd392d5-8760-45f1-9164-4d9c714a6de3") : secret "dns-default-metrics-tls" not found
May 29 01:58:59.101162 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:59.100924    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 01:58:59.101162 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:58:59.100972    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert podName:d21fba59-d15e-4394-bf25-23ba74c209e4 nodeName:}" failed. No retries permitted until 2026-05-29 01:59:01.100957409 +0000 UTC m=+37.683948310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert") pod "ingress-canary-lbjht" (UID: "d21fba59-d15e-4394-bf25-23ba74c209e4") : secret "canary-serving-cert" not found
May 29 01:58:59.130105 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:59.130074    2580 generic.go:358] "Generic (PLEG): container finished" podID="06088ba1-03c3-493f-9665-13dfd7b77b04" containerID="70c83bfb1d39c8c5c0647ea58357ff277628830abbcecc645b76f6e9dc61d2b3" exitCode=0
May 29 01:58:59.130249 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:58:59.130129    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p47b" event={"ID":"06088ba1-03c3-493f-9665-13dfd7b77b04","Type":"ContainerDied","Data":"70c83bfb1d39c8c5c0647ea58357ff277628830abbcecc645b76f6e9dc61d2b3"}
May 29 01:59:00.134521 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:00.134483    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9p47b" event={"ID":"06088ba1-03c3-493f-9665-13dfd7b77b04","Type":"ContainerStarted","Data":"bf1a0cd9fb6ff55dba8d8f767c980620483052a9a999213925278e518167e0fb"}
May 29 01:59:00.177868 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:00.177814    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9p47b" podStartSLOduration=4.93802982 podStartE2EDuration="36.177800229s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:58:26.548888079 +0000 UTC m=+3.131878962" lastFinishedPulling="2026-05-29 01:58:57.788658473 +0000 UTC m=+34.371649371" observedRunningTime="2026-05-29 01:59:00.176229297 +0000 UTC m=+36.759220201" watchObservedRunningTime="2026-05-29 01:59:00.177800229 +0000 UTC m=+36.760791189"
May 29 01:59:01.118082 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:01.117885    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:59:01.118225 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:01.118129    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:59:01.118225 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:01.118031    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 01:59:01.118225 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:01.118212    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert podName:d21fba59-d15e-4394-bf25-23ba74c209e4 nodeName:}" failed. No retries permitted until 2026-05-29 01:59:05.118193646 +0000 UTC m=+41.701184548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert") pod "ingress-canary-lbjht" (UID: "d21fba59-d15e-4394-bf25-23ba74c209e4") : secret "canary-serving-cert" not found
May 29 01:59:01.118419 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:01.118253    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 01:59:01.118419 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:01.118285    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls podName:2cd392d5-8760-45f1-9164-4d9c714a6de3 nodeName:}" failed. No retries permitted until 2026-05-29 01:59:05.118277301 +0000 UTC m=+41.701268190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls") pod "dns-default-fqm6l" (UID: "2cd392d5-8760-45f1-9164-4d9c714a6de3") : secret "dns-default-metrics-tls" not found
May 29 01:59:05.150832 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:05.150801    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:59:05.151205 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:05.150851    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:59:05.151205 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:05.150940    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 01:59:05.151205 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:05.150943    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 01:59:05.151205 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:05.150987    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls podName:2cd392d5-8760-45f1-9164-4d9c714a6de3 nodeName:}" failed. No retries permitted until 2026-05-29 01:59:13.150975231 +0000 UTC m=+49.733966115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls") pod "dns-default-fqm6l" (UID: "2cd392d5-8760-45f1-9164-4d9c714a6de3") : secret "dns-default-metrics-tls" not found
May 29 01:59:05.151205 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:05.151001    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert podName:d21fba59-d15e-4394-bf25-23ba74c209e4 nodeName:}" failed. No retries permitted until 2026-05-29 01:59:13.150994326 +0000 UTC m=+49.733985210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert") pod "ingress-canary-lbjht" (UID: "d21fba59-d15e-4394-bf25-23ba74c209e4") : secret "canary-serving-cert" not found
May 29 01:59:13.207806 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:13.207768    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:59:13.208252 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:13.207829    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:59:13.208252 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:13.207916    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 01:59:13.208252 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:13.207981    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert podName:d21fba59-d15e-4394-bf25-23ba74c209e4 nodeName:}" failed. No retries permitted until 2026-05-29 01:59:29.207964402 +0000 UTC m=+65.790955285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert") pod "ingress-canary-lbjht" (UID: "d21fba59-d15e-4394-bf25-23ba74c209e4") : secret "canary-serving-cert" not found
May 29 01:59:13.208252 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:13.207924    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 01:59:13.208252 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:13.208013    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls podName:2cd392d5-8760-45f1-9164-4d9c714a6de3 nodeName:}" failed. No retries permitted until 2026-05-29 01:59:29.208007554 +0000 UTC m=+65.790998436 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls") pod "dns-default-fqm6l" (UID: "2cd392d5-8760-45f1-9164-4d9c714a6de3") : secret "dns-default-metrics-tls" not found
May 29 01:59:23.123348 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:23.123322    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pktj7"
May 29 01:59:29.219117 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.219072    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 01:59:29.219674 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.219144    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 01:59:29.219674 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:29.219212    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 01:59:29.219674 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:29.219280    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 01:59:29.219674 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:29.219298    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert podName:d21fba59-d15e-4394-bf25-23ba74c209e4 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:01.219281975 +0000 UTC m=+97.802272858 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert") pod "ingress-canary-lbjht" (UID: "d21fba59-d15e-4394-bf25-23ba74c209e4") : secret "canary-serving-cert" not found
May 29 01:59:29.219674 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:29.219345    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls podName:2cd392d5-8760-45f1-9164-4d9c714a6de3 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:01.219328352 +0000 UTC m=+97.802319235 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls") pod "dns-default-fqm6l" (UID: "2cd392d5-8760-45f1-9164-4d9c714a6de3") : secret "dns-default-metrics-tls" not found
May 29 01:59:29.722016 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.721974    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 01:59:29.724802 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.724784    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 29 01:59:29.732838 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:29.732820    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 29 01:59:29.732897 ip-10-0-129-62 kubenswrapper[2580]: E0529 01:59:29.732874    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs podName:7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:33.732859455 +0000 UTC m=+130.315850338 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs") pod "network-metrics-daemon-7475l" (UID: "7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9") : secret "metrics-daemon-secret" not found
May 29 01:59:29.822923 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.822890    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:59:29.826174 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.826158    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 29 01:59:29.835785 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.835766    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 29 01:59:29.847262 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.847234    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt49b\" (UniqueName: \"kubernetes.io/projected/734ad269-957d-44e9-9400-4270c2f494ae-kube-api-access-mt49b\") pod \"network-check-target-qfhm9\" (UID: \"734ad269-957d-44e9-9400-4270c2f494ae\") " pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:59:29.886517 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.886484    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27b64\""
May 29 01:59:29.895299 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:29.895277    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:59:30.029277 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:30.029247    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qfhm9"]
May 29 01:59:30.033507 ip-10-0-129-62 kubenswrapper[2580]: W0529 01:59:30.033474    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734ad269_957d_44e9_9400_4270c2f494ae.slice/crio-cac98706f93a4841d8a8bd660e06b4e1f9c9bc3f69735c8536b50465775b7f04 WatchSource:0}: Error finding container cac98706f93a4841d8a8bd660e06b4e1f9c9bc3f69735c8536b50465775b7f04: Status 404 returned error can't find the container with id cac98706f93a4841d8a8bd660e06b4e1f9c9bc3f69735c8536b50465775b7f04
May 29 01:59:30.190601 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:30.190550    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qfhm9" event={"ID":"734ad269-957d-44e9-9400-4270c2f494ae","Type":"ContainerStarted","Data":"cac98706f93a4841d8a8bd660e06b4e1f9c9bc3f69735c8536b50465775b7f04"}
May 29 01:59:33.197531 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:33.197497    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qfhm9" event={"ID":"734ad269-957d-44e9-9400-4270c2f494ae","Type":"ContainerStarted","Data":"3c9aedbc64df61d226ea1a42dbddedd6e53218bbd04ebfd6361b25afc5ce6a55"}
May 29 01:59:33.197994 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:33.197630    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 01:59:33.233835 ip-10-0-129-62 kubenswrapper[2580]: I0529 01:59:33.233785    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qfhm9" podStartSLOduration=66.613451037 podStartE2EDuration="1m9.233771598s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 01:59:30.035451561 +0000 UTC m=+66.618442457" lastFinishedPulling="2026-05-29 01:59:32.655772119 +0000 UTC m=+69.238763018" observedRunningTime="2026-05-29 01:59:33.216741221 +0000 UTC m=+69.799732121" watchObservedRunningTime="2026-05-29 01:59:33.233771598 +0000 UTC m=+69.816762500"
May 29 02:00:01.234614 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:01.234547    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 02:00:01.234614 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:01.234624    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 02:00:01.235137 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:01.234692    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 02:00:01.235137 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:01.234703    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 02:00:01.235137 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:01.235045    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert podName:d21fba59-d15e-4394-bf25-23ba74c209e4 nodeName:}" failed. No retries permitted until 2026-05-29 02:01:05.234738107 +0000 UTC m=+161.817728989 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert") pod "ingress-canary-lbjht" (UID: "d21fba59-d15e-4394-bf25-23ba74c209e4") : secret "canary-serving-cert" not found
May 29 02:00:01.235290 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:01.235238    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls podName:2cd392d5-8760-45f1-9164-4d9c714a6de3 nodeName:}" failed. No retries permitted until 2026-05-29 02:01:05.235210168 +0000 UTC m=+161.818201065 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls") pod "dns-default-fqm6l" (UID: "2cd392d5-8760-45f1-9164-4d9c714a6de3") : secret "dns-default-metrics-tls" not found
May 29 02:00:04.202206 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:04.202175    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qfhm9"
May 29 02:00:13.576266 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.576235    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb"]
May 29 02:00:13.578951 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.578934    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"]
May 29 02:00:13.579143 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.579099    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb"
May 29 02:00:13.581611 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.581569    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 29 02:00:13.581815 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.581795    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5x6dh\""
May 29 02:00:13.582029 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.582008    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 29 02:00:13.583125 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.583089    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-2zz99"]
May 29 02:00:13.583478 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.583453    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:13.586087 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.586063    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 29 02:00:13.586192 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.586091    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 29 02:00:13.586192 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.586106    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zhcc6\""
May 29 02:00:13.586192 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.586158    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 29 02:00:13.586349 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.586272    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.589697 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.589676    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 29 02:00:13.589766 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.589740    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-sqgp8\""
May 29 02:00:13.590146 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.590126    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 29 02:00:13.590146 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.590145    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 29 02:00:13.590355 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.590134    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 29 02:00:13.595893 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.595017    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"]
May 29 02:00:13.595893 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.595729    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb"]
May 29 02:00:13.596388 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.596368    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-2zz99"]
May 29 02:00:13.597836 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.597818    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 29 02:00:13.614574 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.614550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/146b917f-a47c-45db-a65c-33c5ecaf1237-serving-cert\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.614688 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.614584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cr8w\" (UniqueName: \"kubernetes.io/projected/9b07acd8-0d6c-4c1d-8732-1464c1fba503-kube-api-access-8cr8w\") pod \"volume-data-source-validator-5cf8c89d64-kbzqb\" (UID: \"9b07acd8-0d6c-4c1d-8732-1464c1fba503\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb"
May 29 02:00:13.614688 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.614666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/146b917f-a47c-45db-a65c-33c5ecaf1237-config\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.614758 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.614695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26g88\" (UniqueName: \"kubernetes.io/projected/146b917f-a47c-45db-a65c-33c5ecaf1237-kube-api-access-26g88\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.614758 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.614715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:13.614758 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.614731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gxt\" (UniqueName: \"kubernetes.io/projected/8d030b6a-eb72-47d2-8e07-7f8146110a99-kube-api-access-p2gxt\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:13.614934 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.614819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/146b917f-a47c-45db-a65c-33c5ecaf1237-trusted-ca\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.676439 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.676406    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"]
May 29 02:00:13.679281 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.679260    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.679550 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.679523    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"]
May 29 02:00:13.681569 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.681543    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 29 02:00:13.681684 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.681569    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 29 02:00:13.681684 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.681646    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-th8hb\""
May 29 02:00:13.681837 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.681821    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 29 02:00:13.681943 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.681929    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 29 02:00:13.682107 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.682092    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.683601 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.683567    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b7cd4d9b6-v4s84"]
May 29 02:00:13.686156 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.686135    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.687782 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.687764    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 29 02:00:13.687933 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.687917    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qldgp\""
May 29 02:00:13.688032 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.688015    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 29 02:00:13.688083 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.688019    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 29 02:00:13.688127 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.688019    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 29 02:00:13.688201 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.688185    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zz8s6\""
May 29 02:00:13.688407 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.688387    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 29 02:00:13.688479 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.688451    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 29 02:00:13.688718 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.688703    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 29 02:00:13.688863 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.688847    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 29 02:00:13.688999 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.688979    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 29 02:00:13.689115 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.689101    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 29 02:00:13.691812 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.691793    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"]
May 29 02:00:13.695910 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.695890    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"]
May 29 02:00:13.724611 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.723529    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b7cd4d9b6-v4s84"]
May 29 02:00:13.728902 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.728871    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-26g88\" (UniqueName: \"kubernetes.io/projected/146b917f-a47c-45db-a65c-33c5ecaf1237-kube-api-access-26g88\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.729107 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.729091    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:13.729233 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.729218    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gxt\" (UniqueName: \"kubernetes.io/projected/8d030b6a-eb72-47d2-8e07-7f8146110a99-kube-api-access-p2gxt\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:13.729382 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.729366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37947119-0606-40c1-9ea3-eb3005ce71e3-config\") pod \"kube-storage-version-migrator-operator-565644f49-h7sph\" (UID: \"37947119-0606-40c1-9ea3-eb3005ce71e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.729498 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.729483    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/146b917f-a47c-45db-a65c-33c5ecaf1237-trusted-ca\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.729611 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.729583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzk5\" (UniqueName: \"kubernetes.io/projected/be7b6307-7544-459b-a527-e69e07916e7a-kube-api-access-kjzk5\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.729728 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.729713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5jjl\" (UniqueName: \"kubernetes.io/projected/eedde3d0-971e-40bf-97f8-5c21637f916f-kube-api-access-x5jjl\") pod \"service-ca-operator-57b5b68b8-t6jld\" (UID: \"eedde3d0-971e-40bf-97f8-5c21637f916f\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.729846 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.729831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb9pf\" (UniqueName: \"kubernetes.io/projected/37947119-0606-40c1-9ea3-eb3005ce71e3-kube-api-access-xb9pf\") pod \"kube-storage-version-migrator-operator-565644f49-h7sph\" (UID: \"37947119-0606-40c1-9ea3-eb3005ce71e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.729940 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.729927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.730025 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.730134 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730121    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/146b917f-a47c-45db-a65c-33c5ecaf1237-serving-cert\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.730232 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37947119-0606-40c1-9ea3-eb3005ce71e3-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-h7sph\" (UID: \"37947119-0606-40c1-9ea3-eb3005ce71e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.730321 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-stats-auth\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.730405 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eedde3d0-971e-40bf-97f8-5c21637f916f-config\") pod \"service-ca-operator-57b5b68b8-t6jld\" (UID: \"eedde3d0-971e-40bf-97f8-5c21637f916f\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.730501 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730489    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8cr8w\" (UniqueName: \"kubernetes.io/projected/9b07acd8-0d6c-4c1d-8732-1464c1fba503-kube-api-access-8cr8w\") pod \"volume-data-source-validator-5cf8c89d64-kbzqb\" (UID: \"9b07acd8-0d6c-4c1d-8732-1464c1fba503\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb"
May 29 02:00:13.730616 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-default-certificate\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.730714 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eedde3d0-971e-40bf-97f8-5c21637f916f-serving-cert\") pod \"service-ca-operator-57b5b68b8-t6jld\" (UID: \"eedde3d0-971e-40bf-97f8-5c21637f916f\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.730800 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730778    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/146b917f-a47c-45db-a65c-33c5ecaf1237-trusted-ca\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.730957 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.730819    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/146b917f-a47c-45db-a65c-33c5ecaf1237-config\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.731415 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.731377    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/146b917f-a47c-45db-a65c-33c5ecaf1237-config\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.731671 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:13.729256    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 29 02:00:13.731844 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:13.731834    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls podName:8d030b6a-eb72-47d2-8e07-7f8146110a99 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:14.231813444 +0000 UTC m=+110.814804327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls") pod "cluster-samples-operator-697587599-t5sb7" (UID: "8d030b6a-eb72-47d2-8e07-7f8146110a99") : secret "samples-operator-tls" not found
May 29 02:00:13.736784 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.736764    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/146b917f-a47c-45db-a65c-33c5ecaf1237-serving-cert\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.743513 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.743489    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gxt\" (UniqueName: \"kubernetes.io/projected/8d030b6a-eb72-47d2-8e07-7f8146110a99-kube-api-access-p2gxt\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:13.744641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.744613    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26g88\" (UniqueName: \"kubernetes.io/projected/146b917f-a47c-45db-a65c-33c5ecaf1237-kube-api-access-26g88\") pod \"console-operator-76dff48d59-2zz99\" (UID: \"146b917f-a47c-45db-a65c-33c5ecaf1237\") " pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.744852 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.744830    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cr8w\" (UniqueName: \"kubernetes.io/projected/9b07acd8-0d6c-4c1d-8732-1464c1fba503-kube-api-access-8cr8w\") pod \"volume-data-source-validator-5cf8c89d64-kbzqb\" (UID: \"9b07acd8-0d6c-4c1d-8732-1464c1fba503\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb"
May 29 02:00:13.832000 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.831901    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzk5\" (UniqueName: \"kubernetes.io/projected/be7b6307-7544-459b-a527-e69e07916e7a-kube-api-access-kjzk5\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.832000 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.831942    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jjl\" (UniqueName: \"kubernetes.io/projected/eedde3d0-971e-40bf-97f8-5c21637f916f-kube-api-access-x5jjl\") pod \"service-ca-operator-57b5b68b8-t6jld\" (UID: \"eedde3d0-971e-40bf-97f8-5c21637f916f\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.832000 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.831973    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xb9pf\" (UniqueName: \"kubernetes.io/projected/37947119-0606-40c1-9ea3-eb3005ce71e3-kube-api-access-xb9pf\") pod \"kube-storage-version-migrator-operator-565644f49-h7sph\" (UID: \"37947119-0606-40c1-9ea3-eb3005ce71e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.832000 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.831991    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.832007    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.832029    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37947119-0606-40c1-9ea3-eb3005ce71e3-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-h7sph\" (UID: \"37947119-0606-40c1-9ea3-eb3005ce71e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.832043    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-stats-auth\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.832059    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eedde3d0-971e-40bf-97f8-5c21637f916f-config\") pod \"service-ca-operator-57b5b68b8-t6jld\" (UID: \"eedde3d0-971e-40bf-97f8-5c21637f916f\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.832079    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-default-certificate\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.832094    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eedde3d0-971e-40bf-97f8-5c21637f916f-serving-cert\") pod \"service-ca-operator-57b5b68b8-t6jld\" (UID: \"eedde3d0-971e-40bf-97f8-5c21637f916f\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:13.832154    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:14.332129715 +0000 UTC m=+110.915120616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : configmap references non-existent config key: service-ca.crt
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:13.832199    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.832252    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37947119-0606-40c1-9ea3-eb3005ce71e3-config\") pod \"kube-storage-version-migrator-operator-565644f49-h7sph\" (UID: \"37947119-0606-40c1-9ea3-eb3005ce71e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.832316 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:13.832263    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:14.332246909 +0000 UTC m=+110.915237797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : secret "router-metrics-certs-default" not found
May 29 02:00:13.832853 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.832825    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37947119-0606-40c1-9ea3-eb3005ce71e3-config\") pod \"kube-storage-version-migrator-operator-565644f49-h7sph\" (UID: \"37947119-0606-40c1-9ea3-eb3005ce71e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.832853 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.832840    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eedde3d0-971e-40bf-97f8-5c21637f916f-config\") pod \"service-ca-operator-57b5b68b8-t6jld\" (UID: \"eedde3d0-971e-40bf-97f8-5c21637f916f\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.834500 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.834474    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37947119-0606-40c1-9ea3-eb3005ce71e3-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-h7sph\" (UID: \"37947119-0606-40c1-9ea3-eb3005ce71e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.834676 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.834657    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-default-certificate\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.834752 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.834736    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eedde3d0-971e-40bf-97f8-5c21637f916f-serving-cert\") pod \"service-ca-operator-57b5b68b8-t6jld\" (UID: \"eedde3d0-971e-40bf-97f8-5c21637f916f\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.834895 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.834875    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-stats-auth\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.840027 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.840001    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb9pf\" (UniqueName: \"kubernetes.io/projected/37947119-0606-40c1-9ea3-eb3005ce71e3-kube-api-access-xb9pf\") pod \"kube-storage-version-migrator-operator-565644f49-h7sph\" (UID: \"37947119-0606-40c1-9ea3-eb3005ce71e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:13.840354 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.840324    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzk5\" (UniqueName: \"kubernetes.io/projected/be7b6307-7544-459b-a527-e69e07916e7a-kube-api-access-kjzk5\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:13.840525 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.840509    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5jjl\" (UniqueName: \"kubernetes.io/projected/eedde3d0-971e-40bf-97f8-5c21637f916f-kube-api-access-x5jjl\") pod \"service-ca-operator-57b5b68b8-t6jld\" (UID: \"eedde3d0-971e-40bf-97f8-5c21637f916f\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.892803 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.892778    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb"
May 29 02:00:13.902533 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.902513    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:13.990033 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.990005    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"
May 29 02:00:13.996137 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:13.995960    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"
May 29 02:00:14.021492 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.021444    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb"]
May 29 02:00:14.025207 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:14.025169    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b07acd8_0d6c_4c1d_8732_1464c1fba503.slice/crio-5637cf0702cadc7c3fd5f09b2fa6add613a75d68f0498c71e99f016ab981d936 WatchSource:0}: Error finding container 5637cf0702cadc7c3fd5f09b2fa6add613a75d68f0498c71e99f016ab981d936: Status 404 returned error can't find the container with id 5637cf0702cadc7c3fd5f09b2fa6add613a75d68f0498c71e99f016ab981d936
May 29 02:00:14.042143 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.042098    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-2zz99"]
May 29 02:00:14.045269 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:14.045237    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146b917f_a47c_45db_a65c_33c5ecaf1237.slice/crio-24d4718f213791230e4e2e7fa54e71f90d1d5ccbaf195b69364687af90dd5a40 WatchSource:0}: Error finding container 24d4718f213791230e4e2e7fa54e71f90d1d5ccbaf195b69364687af90dd5a40: Status 404 returned error can't find the container with id 24d4718f213791230e4e2e7fa54e71f90d1d5ccbaf195b69364687af90dd5a40
May 29 02:00:14.121499 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.121467    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld"]
May 29 02:00:14.125648 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:14.125619    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedde3d0_971e_40bf_97f8_5c21637f916f.slice/crio-dc9937e18a273d8f37223d205c9fb3c65f388b562bf59082152571bc7377e439 WatchSource:0}: Error finding container dc9937e18a273d8f37223d205c9fb3c65f388b562bf59082152571bc7377e439: Status 404 returned error can't find the container with id dc9937e18a273d8f37223d205c9fb3c65f388b562bf59082152571bc7377e439
May 29 02:00:14.140113 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.140088    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph"]
May 29 02:00:14.143846 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:14.143817    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37947119_0606_40c1_9ea3_eb3005ce71e3.slice/crio-8e0c80220533d341edd2dc0fac0917abeeb00b0a692c14c4730d9ebffb348e2e WatchSource:0}: Error finding container 8e0c80220533d341edd2dc0fac0917abeeb00b0a692c14c4730d9ebffb348e2e: Status 404 returned error can't find the container with id 8e0c80220533d341edd2dc0fac0917abeeb00b0a692c14c4730d9ebffb348e2e
May 29 02:00:14.235443 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.235415    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:14.235612 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:14.235566    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 29 02:00:14.235653 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:14.235646    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls podName:8d030b6a-eb72-47d2-8e07-7f8146110a99 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:15.235626816 +0000 UTC m=+111.818617710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls") pod "cluster-samples-operator-697587599-t5sb7" (UID: "8d030b6a-eb72-47d2-8e07-7f8146110a99") : secret "samples-operator-tls" not found
May 29 02:00:14.273561 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.273522    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld" event={"ID":"eedde3d0-971e-40bf-97f8-5c21637f916f","Type":"ContainerStarted","Data":"dc9937e18a273d8f37223d205c9fb3c65f388b562bf59082152571bc7377e439"}
May 29 02:00:14.274396 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.274377    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb" event={"ID":"9b07acd8-0d6c-4c1d-8732-1464c1fba503","Type":"ContainerStarted","Data":"5637cf0702cadc7c3fd5f09b2fa6add613a75d68f0498c71e99f016ab981d936"}
May 29 02:00:14.275283 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.275258    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph" event={"ID":"37947119-0606-40c1-9ea3-eb3005ce71e3","Type":"ContainerStarted","Data":"8e0c80220533d341edd2dc0fac0917abeeb00b0a692c14c4730d9ebffb348e2e"}
May 29 02:00:14.276117 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.276090    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" event={"ID":"146b917f-a47c-45db-a65c-33c5ecaf1237","Type":"ContainerStarted","Data":"24d4718f213791230e4e2e7fa54e71f90d1d5ccbaf195b69364687af90dd5a40"}
May 29 02:00:14.336127 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.336095    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:14.336127 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:14.336130    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:14.336295 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:14.336223    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 29 02:00:14.336295 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:14.336257    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:15.33624116 +0000 UTC m=+111.919232043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : configmap references non-existent config key: service-ca.crt
May 29 02:00:14.336295 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:14.336279    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:15.336272894 +0000 UTC m=+111.919263776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : secret "router-metrics-certs-default" not found
May 29 02:00:15.243556 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:15.243490    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:15.244003 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:15.243636    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 29 02:00:15.244003 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:15.243710    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls podName:8d030b6a-eb72-47d2-8e07-7f8146110a99 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:17.243690008 +0000 UTC m=+113.826680894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls") pod "cluster-samples-operator-697587599-t5sb7" (UID: "8d030b6a-eb72-47d2-8e07-7f8146110a99") : secret "samples-operator-tls" not found
May 29 02:00:15.344227 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:15.344189    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:15.344414 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:15.344237    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:15.344414 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:15.344347    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:17.344328265 +0000 UTC m=+113.927319165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : configmap references non-existent config key: service-ca.crt
May 29 02:00:15.344414 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:15.344377    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 29 02:00:15.344580 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:15.344421    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:17.344401589 +0000 UTC m=+113.927392474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : secret "router-metrics-certs-default" not found
May 29 02:00:17.259236 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.259205    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:17.259865 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:17.259837    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 29 02:00:17.260037 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:17.260018    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls podName:8d030b6a-eb72-47d2-8e07-7f8146110a99 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:21.25999727 +0000 UTC m=+117.842988173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls") pod "cluster-samples-operator-697587599-t5sb7" (UID: "8d030b6a-eb72-47d2-8e07-7f8146110a99") : secret "samples-operator-tls" not found
May 29 02:00:17.284112 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.284080    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld" event={"ID":"eedde3d0-971e-40bf-97f8-5c21637f916f","Type":"ContainerStarted","Data":"75e0387cbafd70ef8ac093d285660c629a296bdba1c5a415ba345798ef89b457"}
May 29 02:00:17.285580 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.285554    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb" event={"ID":"9b07acd8-0d6c-4c1d-8732-1464c1fba503","Type":"ContainerStarted","Data":"c5faece33b72f46a19fa16326568b7ac683b7dfd4efe35997aa885ca847a5edd"}
May 29 02:00:17.287245 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.287207    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph" event={"ID":"37947119-0606-40c1-9ea3-eb3005ce71e3","Type":"ContainerStarted","Data":"1f371ddb8a4169d32ff75884a341dd9fad66ac67375fde3b82cad4c78d53b936"}
May 29 02:00:17.289050 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.289028    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" event={"ID":"146b917f-a47c-45db-a65c-33c5ecaf1237","Type":"ContainerStarted","Data":"6c035d506fa2f04212721fc1e2c445642584ad26f80a9f01a8d81c2059058088"}
May 29 02:00:17.289755 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.289735    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:17.291114 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.291094    2580 patch_prober.go:28] interesting pod/console-operator-76dff48d59-2zz99 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.9:8443/readyz\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body=
May 29 02:00:17.291210 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.291185    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" podUID="146b917f-a47c-45db-a65c-33c5ecaf1237" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.9:8443/readyz\": dial tcp 10.134.0.9:8443: connect: connection refused"
May 29 02:00:17.300143 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.300013    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-kbzqb" podStartSLOduration=1.150294205 podStartE2EDuration="4.299997008s" podCreationTimestamp="2026-05-29 02:00:13 +0000 UTC" firstStartedPulling="2026-05-29 02:00:14.027045262 +0000 UTC m=+110.610036145" lastFinishedPulling="2026-05-29 02:00:17.176748065 +0000 UTC m=+113.759738948" observedRunningTime="2026-05-29 02:00:17.29991616 +0000 UTC m=+113.882907066" watchObservedRunningTime="2026-05-29 02:00:17.299997008 +0000 UTC m=+113.882987913"
May 29 02:00:17.314089 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.314034    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph" podStartSLOduration=1.279118558 podStartE2EDuration="4.314013604s" podCreationTimestamp="2026-05-29 02:00:13 +0000 UTC" firstStartedPulling="2026-05-29 02:00:14.14553389 +0000 UTC m=+110.728524773" lastFinishedPulling="2026-05-29 02:00:17.180428929 +0000 UTC m=+113.763419819" observedRunningTime="2026-05-29 02:00:17.312889203 +0000 UTC m=+113.895880108" watchObservedRunningTime="2026-05-29 02:00:17.314013604 +0000 UTC m=+113.897004510"
May 29 02:00:17.328896 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.328633    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" podStartSLOduration=1.199783544 podStartE2EDuration="4.328615174s" podCreationTimestamp="2026-05-29 02:00:13 +0000 UTC" firstStartedPulling="2026-05-29 02:00:14.048499635 +0000 UTC m=+110.631490543" lastFinishedPulling="2026-05-29 02:00:17.177331286 +0000 UTC m=+113.760322173" observedRunningTime="2026-05-29 02:00:17.327530524 +0000 UTC m=+113.910521443" watchObservedRunningTime="2026-05-29 02:00:17.328615174 +0000 UTC m=+113.911606083"
May 29 02:00:17.360630 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.360499    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:17.360630 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:17.360544    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:17.361502 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:17.361079    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:21.36105832 +0000 UTC m=+117.944049209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : configmap references non-existent config key: service-ca.crt
May 29 02:00:17.361502 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:17.361144    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 29 02:00:17.361502 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:17.361177    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:21.361164716 +0000 UTC m=+117.944155608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : secret "router-metrics-certs-default" not found
May 29 02:00:18.293448 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:18.293420    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/0.log"
May 29 02:00:18.293867 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:18.293463    2580 generic.go:358] "Generic (PLEG): container finished" podID="146b917f-a47c-45db-a65c-33c5ecaf1237" containerID="6c035d506fa2f04212721fc1e2c445642584ad26f80a9f01a8d81c2059058088" exitCode=255
May 29 02:00:18.293867 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:18.293574    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" event={"ID":"146b917f-a47c-45db-a65c-33c5ecaf1237","Type":"ContainerDied","Data":"6c035d506fa2f04212721fc1e2c445642584ad26f80a9f01a8d81c2059058088"}
May 29 02:00:18.293867 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:18.293831    2580 scope.go:117] "RemoveContainer" containerID="6c035d506fa2f04212721fc1e2c445642584ad26f80a9f01a8d81c2059058088"
May 29 02:00:18.332406 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:18.332349    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld" podStartSLOduration=2.275713025 podStartE2EDuration="5.332328934s" podCreationTimestamp="2026-05-29 02:00:13 +0000 UTC" firstStartedPulling="2026-05-29 02:00:14.127607675 +0000 UTC m=+110.710598562" lastFinishedPulling="2026-05-29 02:00:17.184223588 +0000 UTC m=+113.767214471" observedRunningTime="2026-05-29 02:00:18.331704565 +0000 UTC m=+114.914695471" watchObservedRunningTime="2026-05-29 02:00:18.332328934 +0000 UTC m=+114.915319841"
May 29 02:00:18.829286 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:18.829257    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kdkn5_ae3f09b5-807b-46e2-a6ea-cb7d47ae16db/dns-node-resolver/0.log"
May 29 02:00:19.298764 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.298732    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:00:19.299157 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.299124    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/0.log"
May 29 02:00:19.299214 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.299158    2580 generic.go:358] "Generic (PLEG): container finished" podID="146b917f-a47c-45db-a65c-33c5ecaf1237" containerID="41795b81257e6d5a4e413c548d95ab5c2f09ab6dd69dceb3aa8adc1e4ae95067" exitCode=255
May 29 02:00:19.299214 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.299192    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" event={"ID":"146b917f-a47c-45db-a65c-33c5ecaf1237","Type":"ContainerDied","Data":"41795b81257e6d5a4e413c548d95ab5c2f09ab6dd69dceb3aa8adc1e4ae95067"}
May 29 02:00:19.299276 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.299220    2580 scope.go:117] "RemoveContainer" containerID="6c035d506fa2f04212721fc1e2c445642584ad26f80a9f01a8d81c2059058088"
May 29 02:00:19.299474 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.299457    2580 scope.go:117] "RemoveContainer" containerID="41795b81257e6d5a4e413c548d95ab5c2f09ab6dd69dceb3aa8adc1e4ae95067"
May 29 02:00:19.299669 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:19.299650    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-2zz99_openshift-console-operator(146b917f-a47c-45db-a65c-33c5ecaf1237)\"" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" podUID="146b917f-a47c-45db-a65c-33c5ecaf1237"
May 29 02:00:19.335982 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.335945    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h"]
May 29 02:00:19.340054 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.340037    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h"
May 29 02:00:19.342308 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.342289    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4cdc5\""
May 29 02:00:19.342408 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.342294    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 29 02:00:19.342408 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.342314    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 29 02:00:19.351245 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.351227    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h"]
May 29 02:00:19.479794 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.479755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jgj\" (UniqueName: \"kubernetes.io/projected/447123ce-921a-49f9-8634-fbcc2d3e0457-kube-api-access-s5jgj\") pod \"migrator-b985f5d89-cgp2h\" (UID: \"447123ce-921a-49f9-8634-fbcc2d3e0457\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h"
May 29 02:00:19.580766 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.580674    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jgj\" (UniqueName: \"kubernetes.io/projected/447123ce-921a-49f9-8634-fbcc2d3e0457-kube-api-access-s5jgj\") pod \"migrator-b985f5d89-cgp2h\" (UID: \"447123ce-921a-49f9-8634-fbcc2d3e0457\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h"
May 29 02:00:19.588544 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.588520    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jgj\" (UniqueName: \"kubernetes.io/projected/447123ce-921a-49f9-8634-fbcc2d3e0457-kube-api-access-s5jgj\") pod \"migrator-b985f5d89-cgp2h\" (UID: \"447123ce-921a-49f9-8634-fbcc2d3e0457\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h"
May 29 02:00:19.648653 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.648616    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h"
May 29 02:00:19.770144 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.770115    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h"]
May 29 02:00:19.773229 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:19.773196    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447123ce_921a_49f9_8634_fbcc2d3e0457.slice/crio-0e731060d2f809574244550703976ba857808385253a602e6fc37bdb260a1181 WatchSource:0}: Error finding container 0e731060d2f809574244550703976ba857808385253a602e6fc37bdb260a1181: Status 404 returned error can't find the container with id 0e731060d2f809574244550703976ba857808385253a602e6fc37bdb260a1181
May 29 02:00:19.828368 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:19.828343    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lmdcx_41a9f0b2-1f52-4fb2-b920-617ce22dc8d0/node-ca/0.log"
May 29 02:00:20.303013 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:20.302987    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:00:20.303432 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:20.303364    2580 scope.go:117] "RemoveContainer" containerID="41795b81257e6d5a4e413c548d95ab5c2f09ab6dd69dceb3aa8adc1e4ae95067"
May 29 02:00:20.303569 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:20.303550    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-2zz99_openshift-console-operator(146b917f-a47c-45db-a65c-33c5ecaf1237)\"" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" podUID="146b917f-a47c-45db-a65c-33c5ecaf1237"
May 29 02:00:20.303965 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:20.303946    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h" event={"ID":"447123ce-921a-49f9-8634-fbcc2d3e0457","Type":"ContainerStarted","Data":"0e731060d2f809574244550703976ba857808385253a602e6fc37bdb260a1181"}
May 29 02:00:21.295966 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:21.295934    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:21.296118 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:21.296098    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 29 02:00:21.296179 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:21.296168    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls podName:8d030b6a-eb72-47d2-8e07-7f8146110a99 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:29.29615307 +0000 UTC m=+125.879143958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls") pod "cluster-samples-operator-697587599-t5sb7" (UID: "8d030b6a-eb72-47d2-8e07-7f8146110a99") : secret "samples-operator-tls" not found
May 29 02:00:21.309335 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:21.309302    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h" event={"ID":"447123ce-921a-49f9-8634-fbcc2d3e0457","Type":"ContainerStarted","Data":"55bc455b5e8a5e8750809d7d08a372598957f652a6c00c0ae648da0aa9a6ca61"}
May 29 02:00:21.309335 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:21.309335    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h" event={"ID":"447123ce-921a-49f9-8634-fbcc2d3e0457","Type":"ContainerStarted","Data":"fe9f423c377a77184aa3dd13f7c97c677773097fb8d0c56746602ce5e6b850f3"}
May 29 02:00:21.396776 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:21.396739    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:21.396776 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:21.396779    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:21.396983 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:21.396915    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 29 02:00:21.396983 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:21.396978    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:29.396963099 +0000 UTC m=+125.979954004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : secret "router-metrics-certs-default" not found
May 29 02:00:21.397058 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:21.397002    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:29.396986097 +0000 UTC m=+125.979976997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : configmap references non-existent config key: service-ca.crt
May 29 02:00:23.903179 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:23.903141    2580 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:23.903565 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:23.903484    2580 scope.go:117] "RemoveContainer" containerID="41795b81257e6d5a4e413c548d95ab5c2f09ab6dd69dceb3aa8adc1e4ae95067"
May 29 02:00:23.903672 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:23.903655    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-2zz99_openshift-console-operator(146b917f-a47c-45db-a65c-33c5ecaf1237)\"" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" podUID="146b917f-a47c-45db-a65c-33c5ecaf1237"
May 29 02:00:27.290311 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:27.290278    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:27.290728 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:27.290712    2580 scope.go:117] "RemoveContainer" containerID="41795b81257e6d5a4e413c548d95ab5c2f09ab6dd69dceb3aa8adc1e4ae95067"
May 29 02:00:27.290900 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:27.290883    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-2zz99_openshift-console-operator(146b917f-a47c-45db-a65c-33c5ecaf1237)\"" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" podUID="146b917f-a47c-45db-a65c-33c5ecaf1237"
May 29 02:00:29.363914 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:29.363864    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:29.366338 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:29.366317    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d030b6a-eb72-47d2-8e07-7f8146110a99-samples-operator-tls\") pod \"cluster-samples-operator-697587599-t5sb7\" (UID: \"8d030b6a-eb72-47d2-8e07-7f8146110a99\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:29.464654 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:29.464582    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:29.464654 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:29.464657    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:29.464896 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:29.464763    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle podName:be7b6307-7544-459b-a527-e69e07916e7a nodeName:}" failed. No retries permitted until 2026-05-29 02:00:45.46474469 +0000 UTC m=+142.047735593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle") pod "router-default-b7cd4d9b6-v4s84" (UID: "be7b6307-7544-459b-a527-e69e07916e7a") : configmap references non-existent config key: service-ca.crt
May 29 02:00:29.466970 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:29.466946    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7b6307-7544-459b-a527-e69e07916e7a-metrics-certs\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:29.499922 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:29.499893    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zhcc6\""
May 29 02:00:29.508558 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:29.508538    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"
May 29 02:00:29.626863 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:29.626769    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-cgp2h" podStartSLOduration=9.523802856 podStartE2EDuration="10.626752235s" podCreationTimestamp="2026-05-29 02:00:19 +0000 UTC" firstStartedPulling="2026-05-29 02:00:19.775529121 +0000 UTC m=+116.358520007" lastFinishedPulling="2026-05-29 02:00:20.878478503 +0000 UTC m=+117.461469386" observedRunningTime="2026-05-29 02:00:21.326077233 +0000 UTC m=+117.909068138" watchObservedRunningTime="2026-05-29 02:00:29.626752235 +0000 UTC m=+126.209743167"
May 29 02:00:29.627360 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:29.627345    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7"]
May 29 02:00:30.333578 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:30.333535    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7" event={"ID":"8d030b6a-eb72-47d2-8e07-7f8146110a99","Type":"ContainerStarted","Data":"7df5159a4ab6cde1e93a2f1bf44a4512b9636dabf462529a9e5375a5024a5350"}
May 29 02:00:32.340319 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:32.340286    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7" event={"ID":"8d030b6a-eb72-47d2-8e07-7f8146110a99","Type":"ContainerStarted","Data":"c2b17fbddfaa25e35adb62c670d34d87f43452052d536bc5aee2204f12aa88ee"}
May 29 02:00:32.340319 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:32.340322    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7" event={"ID":"8d030b6a-eb72-47d2-8e07-7f8146110a99","Type":"ContainerStarted","Data":"c131bfb1a90884e9e3a026b9c673fc1793241f5bdd10f91e519360c37f7f7488"}
May 29 02:00:32.355920 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:32.355881    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-t5sb7" podStartSLOduration=17.332364554 podStartE2EDuration="19.355868757s" podCreationTimestamp="2026-05-29 02:00:13 +0000 UTC" firstStartedPulling="2026-05-29 02:00:29.666230614 +0000 UTC m=+126.249221498" lastFinishedPulling="2026-05-29 02:00:31.689734815 +0000 UTC m=+128.272725701" observedRunningTime="2026-05-29 02:00:32.355510782 +0000 UTC m=+128.938501694" watchObservedRunningTime="2026-05-29 02:00:32.355868757 +0000 UTC m=+128.938859662"
May 29 02:00:33.802826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:33.802786    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 02:00:33.805083 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:33.805050    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9-metrics-certs\") pod \"network-metrics-daemon-7475l\" (UID: \"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9\") " pod="openshift-multus/network-metrics-daemon-7475l"
May 29 02:00:34.081976 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:34.081886    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbkht\""
May 29 02:00:34.090661 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:34.090638    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7475l"
May 29 02:00:34.210948 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:34.210919    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7475l"]
May 29 02:00:34.213503 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:34.213477    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3a7bb8_d8a7_475a_99dc_d8ad9a10bcc9.slice/crio-3f13bdcffe76de2fcd7c34bff0a676f5ec8b61b7335523d0376f680a3a33a441 WatchSource:0}: Error finding container 3f13bdcffe76de2fcd7c34bff0a676f5ec8b61b7335523d0376f680a3a33a441: Status 404 returned error can't find the container with id 3f13bdcffe76de2fcd7c34bff0a676f5ec8b61b7335523d0376f680a3a33a441
May 29 02:00:34.345289 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:34.345201    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7475l" event={"ID":"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9","Type":"ContainerStarted","Data":"3f13bdcffe76de2fcd7c34bff0a676f5ec8b61b7335523d0376f680a3a33a441"}
May 29 02:00:35.348959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:35.348929    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7475l" event={"ID":"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9","Type":"ContainerStarted","Data":"2390fa9d68bf513f7e5f3cf26c82c835b10a777bb420657aff450076bfe3595f"}
May 29 02:00:36.353050 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:36.353017    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7475l" event={"ID":"7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9","Type":"ContainerStarted","Data":"a949b4d861dec54336e3b3bba5281e3d71e4faa203adea2a7cd095c74f518922"}
May 29 02:00:36.367774 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:36.367727    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7475l" podStartSLOduration=131.397567925 podStartE2EDuration="2m12.367714134s" podCreationTimestamp="2026-05-29 01:58:24 +0000 UTC" firstStartedPulling="2026-05-29 02:00:34.215376254 +0000 UTC m=+130.798367137" lastFinishedPulling="2026-05-29 02:00:35.185522464 +0000 UTC m=+131.768513346" observedRunningTime="2026-05-29 02:00:36.366566701 +0000 UTC m=+132.949557607" watchObservedRunningTime="2026-05-29 02:00:36.367714134 +0000 UTC m=+132.950705080"
May 29 02:00:39.973117 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:39.973089    2580 scope.go:117] "RemoveContainer" containerID="41795b81257e6d5a4e413c548d95ab5c2f09ab6dd69dceb3aa8adc1e4ae95067"
May 29 02:00:40.078696 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.078665    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gqztr"]
May 29 02:00:40.081742 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.081719    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.084289 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.084265    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 29 02:00:40.084697 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.084679    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 29 02:00:40.085078 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.085063    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 29 02:00:40.085185 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.085165    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wczvh\""
May 29 02:00:40.085435 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.085421    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 29 02:00:40.096956 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.096932    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gqztr"]
May 29 02:00:40.250382 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.250289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77d7n\" (UniqueName: \"kubernetes.io/projected/d0246d78-bf31-4061-ab18-610154dabef5-kube-api-access-77d7n\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.250382 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.250366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0246d78-bf31-4061-ab18-610154dabef5-crio-socket\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.250569 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.250436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0246d78-bf31-4061-ab18-610154dabef5-data-volume\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.250569 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.250498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0246d78-bf31-4061-ab18-610154dabef5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.250569 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.250559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0246d78-bf31-4061-ab18-610154dabef5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.350981 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.350941    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0246d78-bf31-4061-ab18-610154dabef5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.351169 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.351002    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0246d78-bf31-4061-ab18-610154dabef5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.351169 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.351050    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-77d7n\" (UniqueName: \"kubernetes.io/projected/d0246d78-bf31-4061-ab18-610154dabef5-kube-api-access-77d7n\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.351169 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.351081    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0246d78-bf31-4061-ab18-610154dabef5-crio-socket\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.351169 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.351122    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0246d78-bf31-4061-ab18-610154dabef5-data-volume\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.351365 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.351228    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0246d78-bf31-4061-ab18-610154dabef5-crio-socket\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.351510 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.351489    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0246d78-bf31-4061-ab18-610154dabef5-data-volume\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.351647 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.351625    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0246d78-bf31-4061-ab18-610154dabef5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.353560 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.353539    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0246d78-bf31-4061-ab18-610154dabef5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.359742 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.359724    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77d7n\" (UniqueName: \"kubernetes.io/projected/d0246d78-bf31-4061-ab18-610154dabef5-kube-api-access-77d7n\") pod \"insights-runtime-extractor-gqztr\" (UID: \"d0246d78-bf31-4061-ab18-610154dabef5\") " pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.370571 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.370546    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:00:40.370697 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.370615    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-2zz99" event={"ID":"146b917f-a47c-45db-a65c-33c5ecaf1237","Type":"ContainerStarted","Data":"04bb7f744eb364e774c8f9c01f3f90910ad63f78eaf645d87a671d473d1ca007"}
May 29 02:00:40.370906 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.370888    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:40.390297 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.390276    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gqztr"
May 29 02:00:40.512363 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.512277    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gqztr"]
May 29 02:00:40.515428 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:40.515403    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0246d78_bf31_4061_ab18_610154dabef5.slice/crio-4409546dbed81aff1a418fab35a28fdbaed833cbe6a73926f45aaa4435049469 WatchSource:0}: Error finding container 4409546dbed81aff1a418fab35a28fdbaed833cbe6a73926f45aaa4435049469: Status 404 returned error can't find the container with id 4409546dbed81aff1a418fab35a28fdbaed833cbe6a73926f45aaa4435049469
May 29 02:00:40.592802 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.592773    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-2zz99"
May 29 02:00:40.766650 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.766541    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-j4cnx"]
May 29 02:00:40.769725 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.769703    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-j4cnx"
May 29 02:00:40.771890 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.771873    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 29 02:00:40.771997 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.771873    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 29 02:00:40.771997 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.771926    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5qh48\""
May 29 02:00:40.778573 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.778549    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-j4cnx"]
May 29 02:00:40.956240 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:40.956200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszwp\" (UniqueName: \"kubernetes.io/projected/6f6e5a54-bc55-4252-9f1d-657a5fb7c340-kube-api-access-zszwp\") pod \"downloads-57bd96b888-j4cnx\" (UID: \"6f6e5a54-bc55-4252-9f1d-657a5fb7c340\") " pod="openshift-console/downloads-57bd96b888-j4cnx"
May 29 02:00:41.056835 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:41.056803    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zszwp\" (UniqueName: \"kubernetes.io/projected/6f6e5a54-bc55-4252-9f1d-657a5fb7c340-kube-api-access-zszwp\") pod \"downloads-57bd96b888-j4cnx\" (UID: \"6f6e5a54-bc55-4252-9f1d-657a5fb7c340\") " pod="openshift-console/downloads-57bd96b888-j4cnx"
May 29 02:00:41.063999 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:41.063971    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszwp\" (UniqueName: \"kubernetes.io/projected/6f6e5a54-bc55-4252-9f1d-657a5fb7c340-kube-api-access-zszwp\") pod \"downloads-57bd96b888-j4cnx\" (UID: \"6f6e5a54-bc55-4252-9f1d-657a5fb7c340\") " pod="openshift-console/downloads-57bd96b888-j4cnx"
May 29 02:00:41.078802 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:41.078777    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-j4cnx"
May 29 02:00:41.209248 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:41.209188    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-j4cnx"]
May 29 02:00:41.287641 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:41.287604    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6e5a54_bc55_4252_9f1d_657a5fb7c340.slice/crio-f4d0acb33f02210af012c09b9b4640e0431600b4a03ca27064e36a40975a72a5 WatchSource:0}: Error finding container f4d0acb33f02210af012c09b9b4640e0431600b4a03ca27064e36a40975a72a5: Status 404 returned error can't find the container with id f4d0acb33f02210af012c09b9b4640e0431600b4a03ca27064e36a40975a72a5
May 29 02:00:41.374273 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:41.374238    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-j4cnx" event={"ID":"6f6e5a54-bc55-4252-9f1d-657a5fb7c340","Type":"ContainerStarted","Data":"f4d0acb33f02210af012c09b9b4640e0431600b4a03ca27064e36a40975a72a5"}
May 29 02:00:41.375631 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:41.375603    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqztr" event={"ID":"d0246d78-bf31-4061-ab18-610154dabef5","Type":"ContainerStarted","Data":"e3d67ce5c76dc55e5c82f5dbef3418d342d36181ed1a9017d1999a2c9ccbfce8"}
May 29 02:00:41.375718 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:41.375639    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqztr" event={"ID":"d0246d78-bf31-4061-ab18-610154dabef5","Type":"ContainerStarted","Data":"4409546dbed81aff1a418fab35a28fdbaed833cbe6a73926f45aaa4435049469"}
May 29 02:00:42.381283 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:42.381248    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqztr" event={"ID":"d0246d78-bf31-4061-ab18-610154dabef5","Type":"ContainerStarted","Data":"9a7100f9b392b35a8ddf31dc8ba620c356a7afc415a0ec0fd8188d239c924d3b"}
May 29 02:00:43.385184 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:43.385144    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqztr" event={"ID":"d0246d78-bf31-4061-ab18-610154dabef5","Type":"ContainerStarted","Data":"9b8fcb59ba4c75b71a6af8487d7b40bc26b44bc900aeffe071c361d9abc8f27d"}
May 29 02:00:43.400926 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:43.400826    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gqztr" podStartSLOduration=1.10015553 podStartE2EDuration="3.400809645s" podCreationTimestamp="2026-05-29 02:00:40 +0000 UTC" firstStartedPulling="2026-05-29 02:00:40.580234925 +0000 UTC m=+137.163225823" lastFinishedPulling="2026-05-29 02:00:42.880889042 +0000 UTC m=+139.463879938" observedRunningTime="2026-05-29 02:00:43.399913246 +0000 UTC m=+139.982904151" watchObservedRunningTime="2026-05-29 02:00:43.400809645 +0000 UTC m=+139.983800553"
May 29 02:00:45.490720 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:45.490679    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:45.491271 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:45.491251    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7b6307-7544-459b-a527-e69e07916e7a-service-ca-bundle\") pod \"router-default-b7cd4d9b6-v4s84\" (UID: \"be7b6307-7544-459b-a527-e69e07916e7a\") " pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:45.503403 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:45.503378    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zz8s6\""
May 29 02:00:45.512026 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:45.512007    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:45.641424 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:45.641395    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b7cd4d9b6-v4s84"]
May 29 02:00:45.643944 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:45.643915    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7b6307_7544_459b_a527_e69e07916e7a.slice/crio-563ef90d6b9405f74009dbf114bd324852cec22eee2214adcf4b362aed015d55 WatchSource:0}: Error finding container 563ef90d6b9405f74009dbf114bd324852cec22eee2214adcf4b362aed015d55: Status 404 returned error can't find the container with id 563ef90d6b9405f74009dbf114bd324852cec22eee2214adcf4b362aed015d55
May 29 02:00:46.395395 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:46.395355    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b7cd4d9b6-v4s84" event={"ID":"be7b6307-7544-459b-a527-e69e07916e7a","Type":"ContainerStarted","Data":"bd7705fcedd437e1666efe1f702f8c74103dac786b475ac0f7dfff1a2ed368df"}
May 29 02:00:46.395395 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:46.395399    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b7cd4d9b6-v4s84" event={"ID":"be7b6307-7544-459b-a527-e69e07916e7a","Type":"ContainerStarted","Data":"563ef90d6b9405f74009dbf114bd324852cec22eee2214adcf4b362aed015d55"}
May 29 02:00:46.415606 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:46.415525    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-b7cd4d9b6-v4s84" podStartSLOduration=33.415504822 podStartE2EDuration="33.415504822s" podCreationTimestamp="2026-05-29 02:00:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:00:46.414038933 +0000 UTC m=+142.997029837" watchObservedRunningTime="2026-05-29 02:00:46.415504822 +0000 UTC m=+142.998495730"
May 29 02:00:46.512490 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:46.512455    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:46.515037 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:46.515016    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:47.397761 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:47.397731    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:47.399126 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:47.399101    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b7cd4d9b6-v4s84"
May 29 02:00:47.935023 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:47.934958    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"]
May 29 02:00:47.939179 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:47.939151    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"
May 29 02:00:47.941149 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:47.941124    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 29 02:00:47.941280 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:47.941205    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kv6cb\""
May 29 02:00:47.947962 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:47.947938    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"]
May 29 02:00:48.011762 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:48.011728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94d3dcda-9c06-490b-9c32-ad1cab289bd1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-m25c4\" (UID: \"94d3dcda-9c06-490b-9c32-ad1cab289bd1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"
May 29 02:00:48.112995 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:48.112962    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94d3dcda-9c06-490b-9c32-ad1cab289bd1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-m25c4\" (UID: \"94d3dcda-9c06-490b-9c32-ad1cab289bd1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"
May 29 02:00:48.116077 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:48.116046    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94d3dcda-9c06-490b-9c32-ad1cab289bd1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-m25c4\" (UID: \"94d3dcda-9c06-490b-9c32-ad1cab289bd1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"
May 29 02:00:48.250082 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:48.250001    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"
May 29 02:00:48.389688 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:48.389645    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"]
May 29 02:00:48.393133 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:48.393084    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d3dcda_9c06_490b_9c32_ad1cab289bd1.slice/crio-653d1ee04634f74024f163914887071999cb5101986048ff911abcb99860bc12 WatchSource:0}: Error finding container 653d1ee04634f74024f163914887071999cb5101986048ff911abcb99860bc12: Status 404 returned error can't find the container with id 653d1ee04634f74024f163914887071999cb5101986048ff911abcb99860bc12
May 29 02:00:48.401242 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:48.401217    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4" event={"ID":"94d3dcda-9c06-490b-9c32-ad1cab289bd1","Type":"ContainerStarted","Data":"653d1ee04634f74024f163914887071999cb5101986048ff911abcb99860bc12"}
May 29 02:00:50.410549 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.410512    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4" event={"ID":"94d3dcda-9c06-490b-9c32-ad1cab289bd1","Type":"ContainerStarted","Data":"08cbf5060b284c8b211cf20c704d1179d235e108b3703a270408d102a294b94b"}
May 29 02:00:50.411013 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.410821    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"
May 29 02:00:50.417411 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.417039    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4"
May 29 02:00:50.427584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.427529    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-m25c4" podStartSLOduration=2.230778628 podStartE2EDuration="3.427512427s" podCreationTimestamp="2026-05-29 02:00:47 +0000 UTC" firstStartedPulling="2026-05-29 02:00:48.395235462 +0000 UTC m=+144.978226349" lastFinishedPulling="2026-05-29 02:00:49.591969245 +0000 UTC m=+146.174960148" observedRunningTime="2026-05-29 02:00:50.426767286 +0000 UTC m=+147.009758192" watchObservedRunningTime="2026-05-29 02:00:50.427512427 +0000 UTC m=+147.010503333"
May 29 02:00:50.496326 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.496295    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-67c96447c9-cfktq"]
May 29 02:00:50.523942 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.523911    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c96447c9-cfktq"]
May 29 02:00:50.524118 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.524067    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.527085 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.526992    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 29 02:00:50.527085 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.526997    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 29 02:00:50.527288 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.526999    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 29 02:00:50.527288 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.527008    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8pztx\""
May 29 02:00:50.527288 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.527119    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 29 02:00:50.527444 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.527313    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 29 02:00:50.636001 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.635961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-oauth-config\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.636165 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.636014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-config\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.636165 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.636035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-serving-cert\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.636165 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.636086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-service-ca\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.636165 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.636131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-oauth-serving-cert\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.636397 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.636171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw78j\" (UniqueName: \"kubernetes.io/projected/adc062cf-dc64-4df1-b444-d01cb21b4e47-kube-api-access-jw78j\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.737604 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.737499    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-oauth-config\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.737604 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.737544    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-config\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.737814 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.737731    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-serving-cert\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.737814 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.737771    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-service-ca\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.737814 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.737797    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-oauth-serving-cert\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.737957 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.737831    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jw78j\" (UniqueName: \"kubernetes.io/projected/adc062cf-dc64-4df1-b444-d01cb21b4e47-kube-api-access-jw78j\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.738298 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.738264    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-config\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.738768 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.738745    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-service-ca\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.738868 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.738774    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-oauth-serving-cert\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.740278 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.740260    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-oauth-config\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.745744 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.745723    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw78j\" (UniqueName: \"kubernetes.io/projected/adc062cf-dc64-4df1-b444-d01cb21b4e47-kube-api-access-jw78j\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.749736 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.749711    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-serving-cert\") pod \"console-67c96447c9-cfktq\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") " pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.836179 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.836139    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:00:50.978940 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:50.978816    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c96447c9-cfktq"]
May 29 02:00:50.981668 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:50.981550    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc062cf_dc64_4df1_b444_d01cb21b4e47.slice/crio-348759d4c0f5e4e7281fb31a827ce5e9a5b4adf3e0a753a4ad812740f711fd86 WatchSource:0}: Error finding container 348759d4c0f5e4e7281fb31a827ce5e9a5b4adf3e0a753a4ad812740f711fd86: Status 404 returned error can't find the container with id 348759d4c0f5e4e7281fb31a827ce5e9a5b4adf3e0a753a4ad812740f711fd86
May 29 02:00:51.006648 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.006549    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"]
May 29 02:00:51.026186 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.026154    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"]
May 29 02:00:51.026335 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.026306    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.028725 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.028696    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 29 02:00:51.028832 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.028725    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 29 02:00:51.028894 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.028872    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vjvpf\""
May 29 02:00:51.029048 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.028964    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 29 02:00:51.029142 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.029066    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 29 02:00:51.029392 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.029373    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 29 02:00:51.141991 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.141932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.142168 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.142004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a198136c-fc63-42cb-9bb2-ae343cbad7c2-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.142168 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.142102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.142168 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.142163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xr7\" (UniqueName: \"kubernetes.io/projected/a198136c-fc63-42cb-9bb2-ae343cbad7c2-kube-api-access-84xr7\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.243034 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.243000    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.243212 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.243065    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a198136c-fc63-42cb-9bb2-ae343cbad7c2-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.243212 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:51.243124    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 29 02:00:51.243212 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.243142    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.243212 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:51.243200    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-tls podName:a198136c-fc63-42cb-9bb2-ae343cbad7c2 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:51.743179014 +0000 UTC m=+148.326169919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-tdsdc" (UID: "a198136c-fc63-42cb-9bb2-ae343cbad7c2") : secret "prometheus-operator-tls" not found
May 29 02:00:51.243423 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.243246    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-84xr7\" (UniqueName: \"kubernetes.io/projected/a198136c-fc63-42cb-9bb2-ae343cbad7c2-kube-api-access-84xr7\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.243773 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.243745    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a198136c-fc63-42cb-9bb2-ae343cbad7c2-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.245762 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.245743    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.251495 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.251474    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xr7\" (UniqueName: \"kubernetes.io/projected/a198136c-fc63-42cb-9bb2-ae343cbad7c2-kube-api-access-84xr7\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.414739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.414696    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c96447c9-cfktq" event={"ID":"adc062cf-dc64-4df1-b444-d01cb21b4e47","Type":"ContainerStarted","Data":"348759d4c0f5e4e7281fb31a827ce5e9a5b4adf3e0a753a4ad812740f711fd86"}
May 29 02:00:51.748012 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:51.747926    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:51.748155 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:51.748124    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 29 02:00:51.748219 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:00:51.748207    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-tls podName:a198136c-fc63-42cb-9bb2-ae343cbad7c2 nodeName:}" failed. No retries permitted until 2026-05-29 02:00:52.748184721 +0000 UTC m=+149.331175626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-tdsdc" (UID: "a198136c-fc63-42cb-9bb2-ae343cbad7c2") : secret "prometheus-operator-tls" not found
May 29 02:00:52.758374 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:52.758337    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:52.760881 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:52.760854    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a198136c-fc63-42cb-9bb2-ae343cbad7c2-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-tdsdc\" (UID: \"a198136c-fc63-42cb-9bb2-ae343cbad7c2\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:52.837898 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:52.837861    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"
May 29 02:00:58.246800 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:58.246776    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc"]
May 29 02:00:58.255688 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:00:58.255657    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda198136c_fc63_42cb_9bb2_ae343cbad7c2.slice/crio-4d0816a9f7de9e0d4d09a6044284085ecf5f76a6b6c5750a1717f8284a25ad9a WatchSource:0}: Error finding container 4d0816a9f7de9e0d4d09a6044284085ecf5f76a6b6c5750a1717f8284a25ad9a: Status 404 returned error can't find the container with id 4d0816a9f7de9e0d4d09a6044284085ecf5f76a6b6c5750a1717f8284a25ad9a
May 29 02:00:58.440292 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:58.440213    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-j4cnx" event={"ID":"6f6e5a54-bc55-4252-9f1d-657a5fb7c340","Type":"ContainerStarted","Data":"d70ae3e22dd679fbcde393115647b8f8cb1f05acdbd0f9eec700dd04ed9c6d77"}
May 29 02:00:58.440531 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:58.440510    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-j4cnx"
May 29 02:00:58.441540 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:58.441513    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc" event={"ID":"a198136c-fc63-42cb-9bb2-ae343cbad7c2","Type":"ContainerStarted","Data":"4d0816a9f7de9e0d4d09a6044284085ecf5f76a6b6c5750a1717f8284a25ad9a"}
May 29 02:00:58.441853 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:58.441830    2580 patch_prober.go:28] interesting pod/downloads-57bd96b888-j4cnx container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" start-of-body=
May 29 02:00:58.441948 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:58.441875    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-57bd96b888-j4cnx" podUID="6f6e5a54-bc55-4252-9f1d-657a5fb7c340" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused"
May 29 02:00:58.458900 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:58.458835    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-j4cnx" podStartSLOduration=1.526913008 podStartE2EDuration="18.458818159s" podCreationTimestamp="2026-05-29 02:00:40 +0000 UTC" firstStartedPulling="2026-05-29 02:00:41.289896727 +0000 UTC m=+137.872887611" lastFinishedPulling="2026-05-29 02:00:58.221801873 +0000 UTC m=+154.804792762" observedRunningTime="2026-05-29 02:00:58.457256709 +0000 UTC m=+155.040247616" watchObservedRunningTime="2026-05-29 02:00:58.458818159 +0000 UTC m=+155.041809065"
May 29 02:00:59.457002 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:00:59.456865    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-j4cnx"
May 29 02:01:00.348538 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:01:00.348492    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-lbjht" podUID="d21fba59-d15e-4394-bf25-23ba74c209e4"
May 29 02:01:00.358839 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:01:00.358789    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fqm6l" podUID="2cd392d5-8760-45f1-9164-4d9c714a6de3"
May 29 02:01:00.451632 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:00.451411    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc" event={"ID":"a198136c-fc63-42cb-9bb2-ae343cbad7c2","Type":"ContainerStarted","Data":"96ffbb091c39b10642923fb7b2f2954787ba69e13e0772b8c45fde859b01cdb5"}
May 29 02:01:00.451632 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:00.451430    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 02:01:02.459231 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:02.459191    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc" event={"ID":"a198136c-fc63-42cb-9bb2-ae343cbad7c2","Type":"ContainerStarted","Data":"beb071af600d17c06b50d75e693fd98c6b240118eeb08d44825401a1943e2129"}
May 29 02:01:02.460815 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:02.460783    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c96447c9-cfktq" event={"ID":"adc062cf-dc64-4df1-b444-d01cb21b4e47","Type":"ContainerStarted","Data":"c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd"}
May 29 02:01:02.477412 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:02.477361    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tdsdc" podStartSLOduration=10.752179902 podStartE2EDuration="12.47734854s" podCreationTimestamp="2026-05-29 02:00:50 +0000 UTC" firstStartedPulling="2026-05-29 02:00:58.257638578 +0000 UTC m=+154.840629474" lastFinishedPulling="2026-05-29 02:00:59.982807226 +0000 UTC m=+156.565798112" observedRunningTime="2026-05-29 02:01:02.475532367 +0000 UTC m=+159.058523271" watchObservedRunningTime="2026-05-29 02:01:02.47734854 +0000 UTC m=+159.060339445"
May 29 02:01:02.492143 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:02.492084    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-67c96447c9-cfktq" podStartSLOduration=1.558381832 podStartE2EDuration="12.492071089s" podCreationTimestamp="2026-05-29 02:00:50 +0000 UTC" firstStartedPulling="2026-05-29 02:00:50.983766086 +0000 UTC m=+147.566756970" lastFinishedPulling="2026-05-29 02:01:01.917455329 +0000 UTC m=+158.500446227" observedRunningTime="2026-05-29 02:01:02.49144271 +0000 UTC m=+159.074433628" watchObservedRunningTime="2026-05-29 02:01:02.492071089 +0000 UTC m=+159.075061997"
May 29 02:01:04.423913 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.423875    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s9kqs"]
May 29 02:01:04.454147 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.454114    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.456161 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.456122    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 29 02:01:04.456161 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.456138    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jwsv6\""
May 29 02:01:04.456374 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.456142    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 29 02:01:04.456374 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.456238    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 29 02:01:04.571053 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.571009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da4ada43-7e96-4292-912b-d4c4d45d48c8-metrics-client-ca\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.571249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.571066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.571249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.571091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-wtmp\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.571249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.571140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da4ada43-7e96-4292-912b-d4c4d45d48c8-root\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.571249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.571176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-tls\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.571249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.571193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.571249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.571213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpnrd\" (UniqueName: \"kubernetes.io/projected/da4ada43-7e96-4292-912b-d4c4d45d48c8-kube-api-access-qpnrd\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.571249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.571238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-textfile\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.571547 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.571299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da4ada43-7e96-4292-912b-d4c4d45d48c8-sys\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.672495 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672451    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-tls\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.672495 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672504    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.672816 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672531    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qpnrd\" (UniqueName: \"kubernetes.io/projected/da4ada43-7e96-4292-912b-d4c4d45d48c8-kube-api-access-qpnrd\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.672816 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672557    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-textfile\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.672816 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672623    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da4ada43-7e96-4292-912b-d4c4d45d48c8-sys\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.672816 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672676    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da4ada43-7e96-4292-912b-d4c4d45d48c8-metrics-client-ca\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.672816 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672719    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.672816 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672751    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-wtmp\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.672816 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672810    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da4ada43-7e96-4292-912b-d4c4d45d48c8-root\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.673207 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672886    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da4ada43-7e96-4292-912b-d4c4d45d48c8-root\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.673207 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672968    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-textfile\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.673207 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.672749    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da4ada43-7e96-4292-912b-d4c4d45d48c8-sys\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.673207 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.673027    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-wtmp\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.673430 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.673403    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da4ada43-7e96-4292-912b-d4c4d45d48c8-metrics-client-ca\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.673499 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.673454    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.675546 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.675492    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-tls\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.675661 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.675559    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da4ada43-7e96-4292-912b-d4c4d45d48c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.681881 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.681855    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpnrd\" (UniqueName: \"kubernetes.io/projected/da4ada43-7e96-4292-912b-d4c4d45d48c8-kube-api-access-qpnrd\") pod \"node-exporter-s9kqs\" (UID: \"da4ada43-7e96-4292-912b-d4c4d45d48c8\") " pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.765124 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:04.765080    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s9kqs"
May 29 02:01:04.774859 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:01:04.774823    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4ada43_7e96_4292_912b_d4c4d45d48c8.slice/crio-a11184e08e33f29dc0dc19f2cf5d13aa7e138ca7743860e70ad0b2943f1c3385 WatchSource:0}: Error finding container a11184e08e33f29dc0dc19f2cf5d13aa7e138ca7743860e70ad0b2943f1c3385: Status 404 returned error can't find the container with id a11184e08e33f29dc0dc19f2cf5d13aa7e138ca7743860e70ad0b2943f1c3385
May 29 02:01:05.278752 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:05.278711    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 02:01:05.279041 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:05.278764    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 02:01:05.281473 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:05.281449    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cd392d5-8760-45f1-9164-4d9c714a6de3-metrics-tls\") pod \"dns-default-fqm6l\" (UID: \"2cd392d5-8760-45f1-9164-4d9c714a6de3\") " pod="openshift-dns/dns-default-fqm6l"
May 29 02:01:05.281621 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:05.281569    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21fba59-d15e-4394-bf25-23ba74c209e4-cert\") pod \"ingress-canary-lbjht\" (UID: \"d21fba59-d15e-4394-bf25-23ba74c209e4\") " pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 02:01:05.470542 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:05.470502    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s9kqs" event={"ID":"da4ada43-7e96-4292-912b-d4c4d45d48c8","Type":"ContainerStarted","Data":"a11184e08e33f29dc0dc19f2cf5d13aa7e138ca7743860e70ad0b2943f1c3385"}
May 29 02:01:05.554509 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:05.554461    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fmgpb\""
May 29 02:01:05.563216 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:05.563181    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lbjht"
May 29 02:01:05.729188 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:05.729130    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lbjht"]
May 29 02:01:05.734091 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:01:05.734058    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21fba59_d15e_4394_bf25_23ba74c209e4.slice/crio-2fcdeee682e3571e9ba61808641d2948e10945e617803946b350885e5faad102 WatchSource:0}: Error finding container 2fcdeee682e3571e9ba61808641d2948e10945e617803946b350885e5faad102: Status 404 returned error can't find the container with id 2fcdeee682e3571e9ba61808641d2948e10945e617803946b350885e5faad102
May 29 02:01:06.475457 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:06.475426    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lbjht" event={"ID":"d21fba59-d15e-4394-bf25-23ba74c209e4","Type":"ContainerStarted","Data":"2fcdeee682e3571e9ba61808641d2948e10945e617803946b350885e5faad102"}
May 29 02:01:07.479940 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:07.479900    2580 generic.go:358] "Generic (PLEG): container finished" podID="da4ada43-7e96-4292-912b-d4c4d45d48c8" containerID="d28cd8bd446ac6a269d6bb32ff9832bfb8d46cff8cce0722fcb149a84c742449" exitCode=0
May 29 02:01:07.480344 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:07.479981    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s9kqs" event={"ID":"da4ada43-7e96-4292-912b-d4c4d45d48c8","Type":"ContainerDied","Data":"d28cd8bd446ac6a269d6bb32ff9832bfb8d46cff8cce0722fcb149a84c742449"}
May 29 02:01:08.484006 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.483975    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s9kqs" event={"ID":"da4ada43-7e96-4292-912b-d4c4d45d48c8","Type":"ContainerStarted","Data":"8e1aeb63ed6cf67b61f28f8dce3a7275cdab1da11a8fb6a6f8cce9e7e3c4d8c7"}
May 29 02:01:08.484308 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.484015    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s9kqs" event={"ID":"da4ada43-7e96-4292-912b-d4c4d45d48c8","Type":"ContainerStarted","Data":"5dc7b0c9c9fb6c362363c40ff2808fc2b3e049b65465d7705a9413a47be577d0"}
May 29 02:01:08.521458 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.521414    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s9kqs" podStartSLOduration=2.571357625 podStartE2EDuration="4.52139841s" podCreationTimestamp="2026-05-29 02:01:04 +0000 UTC" firstStartedPulling="2026-05-29 02:01:04.776845909 +0000 UTC m=+161.359836793" lastFinishedPulling="2026-05-29 02:01:06.726886679 +0000 UTC m=+163.309877578" observedRunningTime="2026-05-29 02:01:08.518083918 +0000 UTC m=+165.101074822" watchObservedRunningTime="2026-05-29 02:01:08.52139841 +0000 UTC m=+165.104389316"
May 29 02:01:08.675467 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.675433    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"]
May 29 02:01:08.695339 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.695260    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"]
May 29 02:01:08.695491 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.695424    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.697871 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.697843    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 29 02:01:08.698009 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.697913    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a68eoirvaccb7\""
May 29 02:01:08.698009 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.697990    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 29 02:01:08.698128 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.697843    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 29 02:01:08.698286 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.698259    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nkjcv\""
May 29 02:01:08.698416 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.698398    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 29 02:01:08.813323 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.813292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebb96298-ed02-4b73-9224-52a8e4eb19cb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.813498 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.813332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebb96298-ed02-4b73-9224-52a8e4eb19cb-secret-metrics-server-client-certs\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.813498 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.813358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebb96298-ed02-4b73-9224-52a8e4eb19cb-client-ca-bundle\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.813498 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.813416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpznx\" (UniqueName: \"kubernetes.io/projected/ebb96298-ed02-4b73-9224-52a8e4eb19cb-kube-api-access-rpznx\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.813498 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.813451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebb96298-ed02-4b73-9224-52a8e4eb19cb-audit-log\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.813498 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.813471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebb96298-ed02-4b73-9224-52a8e4eb19cb-metrics-server-audit-profiles\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.813727 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.813527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebb96298-ed02-4b73-9224-52a8e4eb19cb-secret-metrics-server-tls\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.914410 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.914371    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebb96298-ed02-4b73-9224-52a8e4eb19cb-secret-metrics-server-tls\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.914583 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.914426    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebb96298-ed02-4b73-9224-52a8e4eb19cb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.914661 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.914630    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebb96298-ed02-4b73-9224-52a8e4eb19cb-secret-metrics-server-client-certs\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.914703 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.914682    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebb96298-ed02-4b73-9224-52a8e4eb19cb-client-ca-bundle\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.914760 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.914736    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rpznx\" (UniqueName: \"kubernetes.io/projected/ebb96298-ed02-4b73-9224-52a8e4eb19cb-kube-api-access-rpznx\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.914833 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.914818    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebb96298-ed02-4b73-9224-52a8e4eb19cb-audit-log\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.914881 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.914856    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebb96298-ed02-4b73-9224-52a8e4eb19cb-metrics-server-audit-profiles\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.915256 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.915216    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebb96298-ed02-4b73-9224-52a8e4eb19cb-audit-log\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.915376 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.915316    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebb96298-ed02-4b73-9224-52a8e4eb19cb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.915856 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.915833    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebb96298-ed02-4b73-9224-52a8e4eb19cb-metrics-server-audit-profiles\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.916960 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.916930    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebb96298-ed02-4b73-9224-52a8e4eb19cb-secret-metrics-server-tls\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.917117 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.917097    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebb96298-ed02-4b73-9224-52a8e4eb19cb-secret-metrics-server-client-certs\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.917206 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.917188    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebb96298-ed02-4b73-9224-52a8e4eb19cb-client-ca-bundle\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:08.922310 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:08.922290    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpznx\" (UniqueName: \"kubernetes.io/projected/ebb96298-ed02-4b73-9224-52a8e4eb19cb-kube-api-access-rpznx\") pod \"metrics-server-bb8bb4fc7-2n67j\" (UID: \"ebb96298-ed02-4b73-9224-52a8e4eb19cb\") " pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:09.005913 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:09.005872    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:09.142529 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:09.142492    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"]
May 29 02:01:09.145508 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:01:09.145477    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb96298_ed02_4b73_9224_52a8e4eb19cb.slice/crio-c57fefd7477eba99722e2b45fa3e01e0e6339ecc6209c25e71f94cbd01bb2440 WatchSource:0}: Error finding container c57fefd7477eba99722e2b45fa3e01e0e6339ecc6209c25e71f94cbd01bb2440: Status 404 returned error can't find the container with id c57fefd7477eba99722e2b45fa3e01e0e6339ecc6209c25e71f94cbd01bb2440
May 29 02:01:09.487829 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:09.487757    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j" event={"ID":"ebb96298-ed02-4b73-9224-52a8e4eb19cb","Type":"ContainerStarted","Data":"c57fefd7477eba99722e2b45fa3e01e0e6339ecc6209c25e71f94cbd01bb2440"}
May 29 02:01:09.489142 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:09.489113    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lbjht" event={"ID":"d21fba59-d15e-4394-bf25-23ba74c209e4","Type":"ContainerStarted","Data":"d571905e75f0ceef95e4ee8a8696824e98961967de87b3e454ff13e37a1b62b5"}
May 29 02:01:09.503687 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:09.503622    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lbjht" podStartSLOduration=129.762238921 podStartE2EDuration="2m12.503603114s" podCreationTimestamp="2026-05-29 01:58:57 +0000 UTC" firstStartedPulling="2026-05-29 02:01:05.737246567 +0000 UTC m=+162.320237451" lastFinishedPulling="2026-05-29 02:01:08.478610738 +0000 UTC m=+165.061601644" observedRunningTime="2026-05-29 02:01:09.503277293 +0000 UTC m=+166.086268198" watchObservedRunningTime="2026-05-29 02:01:09.503603114 +0000 UTC m=+166.086594011"
May 29 02:01:10.142608 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.142554    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c96447c9-cfktq"]
May 29 02:01:10.286583 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.286548    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-84cddbdcf5-hfwq8"]
May 29 02:01:10.312193 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.311519    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cddbdcf5-hfwq8"]
May 29 02:01:10.312193 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.311694    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.325903 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.325879    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 29 02:01:10.428665 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.428535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pthw\" (UniqueName: \"kubernetes.io/projected/c537270a-1d71-431e-ace3-385999191fa4-kube-api-access-4pthw\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.428920 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.428893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-trusted-ca-bundle\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.429017 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.428952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-service-ca\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.429081 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.429055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-serving-cert\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.429139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.429104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-console-config\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.429139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.429121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-oauth-serving-cert\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.429234 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.429141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-oauth-config\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.530544 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.530508    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4pthw\" (UniqueName: \"kubernetes.io/projected/c537270a-1d71-431e-ace3-385999191fa4-kube-api-access-4pthw\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.530544 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.530548    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-trusted-ca-bundle\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.531045 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.530584    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-service-ca\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.531045 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.530639    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-serving-cert\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.531045 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.530664    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-console-config\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.531045 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.530685    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-oauth-serving-cert\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.531045 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.530709    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-oauth-config\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.531481 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.531463    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-console-config\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.535638 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.535610    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-trusted-ca-bundle\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.543662 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.543611    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-service-ca\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.551569 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.551544    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-oauth-serving-cert\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.560062 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.560034    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-serving-cert\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.560062 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.560052    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-oauth-config\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.560231 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.560207    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pthw\" (UniqueName: \"kubernetes.io/projected/c537270a-1d71-431e-ace3-385999191fa4-kube-api-access-4pthw\") pod \"console-84cddbdcf5-hfwq8\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") " pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.621390 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.621353    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:10.786351 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.786313    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cddbdcf5-hfwq8"]
May 29 02:01:10.790522 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:01:10.790494    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc537270a_1d71_431e_ace3_385999191fa4.slice/crio-3ea9bcb1f046111cd283fa3ed148e9c40fbbbde9acea74ae94dcea09fd823493 WatchSource:0}: Error finding container 3ea9bcb1f046111cd283fa3ed148e9c40fbbbde9acea74ae94dcea09fd823493: Status 404 returned error can't find the container with id 3ea9bcb1f046111cd283fa3ed148e9c40fbbbde9acea74ae94dcea09fd823493
May 29 02:01:10.836423 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:10.836397    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:01:11.496739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:11.496699    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cddbdcf5-hfwq8" event={"ID":"c537270a-1d71-431e-ace3-385999191fa4","Type":"ContainerStarted","Data":"bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0"}
May 29 02:01:11.496739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:11.496740    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cddbdcf5-hfwq8" event={"ID":"c537270a-1d71-431e-ace3-385999191fa4","Type":"ContainerStarted","Data":"3ea9bcb1f046111cd283fa3ed148e9c40fbbbde9acea74ae94dcea09fd823493"}
May 29 02:01:11.516298 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:11.516251    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-84cddbdcf5-hfwq8" podStartSLOduration=1.5162370630000002 podStartE2EDuration="1.516237063s" podCreationTimestamp="2026-05-29 02:01:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:01:11.514863962 +0000 UTC m=+168.097854868" watchObservedRunningTime="2026-05-29 02:01:11.516237063 +0000 UTC m=+168.099227967"
May 29 02:01:13.503866 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:13.503825    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j" event={"ID":"ebb96298-ed02-4b73-9224-52a8e4eb19cb","Type":"ContainerStarted","Data":"b176c79a4cf2ff547443f64c45151482341649b959e9593129c6832c313c9049"}
May 29 02:01:13.519144 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:13.519088    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j" podStartSLOduration=1.703215803 podStartE2EDuration="5.519069037s" podCreationTimestamp="2026-05-29 02:01:08 +0000 UTC" firstStartedPulling="2026-05-29 02:01:09.147395046 +0000 UTC m=+165.730385945" lastFinishedPulling="2026-05-29 02:01:12.963248297 +0000 UTC m=+169.546239179" observedRunningTime="2026-05-29 02:01:13.518329465 +0000 UTC m=+170.101320369" watchObservedRunningTime="2026-05-29 02:01:13.519069037 +0000 UTC m=+170.102059944"
May 29 02:01:14.863519 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:14.863481    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84cddbdcf5-hfwq8"]
May 29 02:01:14.900100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:14.900063    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-758c75c9c6-2468d"]
May 29 02:01:14.934657 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:14.934619    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758c75c9c6-2468d"]
May 29 02:01:14.934805 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:14.934731    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:14.970340 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:14.970310    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fqm6l"
May 29 02:01:14.972430 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:14.972408    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7t46n\""
May 29 02:01:14.981017 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:14.981000    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fqm6l"
May 29 02:01:15.071267 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.071233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-console-config\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.071431 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.071397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-trusted-ca-bundle\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.071495 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.071483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-service-ca\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.071549 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.071508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-oauth-config\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.071549 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.071523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-oauth-serving-cert\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.071549 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.071543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgp59\" (UniqueName: \"kubernetes.io/projected/e57ce052-bf4e-4667-a732-7601bdee84bf-kube-api-access-fgp59\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.071714 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.071655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-serving-cert\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.107833 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.107799    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fqm6l"]
May 29 02:01:15.111258 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:01:15.111231    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd392d5_8760_45f1_9164_4d9c714a6de3.slice/crio-87d9eb0854259312390bf0c9ad03bb7cf9690e9d1b583d9ecca7e08484ff5ae5 WatchSource:0}: Error finding container 87d9eb0854259312390bf0c9ad03bb7cf9690e9d1b583d9ecca7e08484ff5ae5: Status 404 returned error can't find the container with id 87d9eb0854259312390bf0c9ad03bb7cf9690e9d1b583d9ecca7e08484ff5ae5
May 29 02:01:15.172702 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.172657    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-service-ca\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.172702 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.172704    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-oauth-config\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.172945 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.172723    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-oauth-serving-cert\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.172945 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.172740    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fgp59\" (UniqueName: \"kubernetes.io/projected/e57ce052-bf4e-4667-a732-7601bdee84bf-kube-api-access-fgp59\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.172945 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.172774    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-serving-cert\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.172945 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.172800    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-console-config\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.173149 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.173110    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-trusted-ca-bundle\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.173446 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.173423    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-service-ca\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.173744 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.173722    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-oauth-serving-cert\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.173806 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.173744    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-console-config\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.174249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.174232    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-trusted-ca-bundle\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.175330 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.175308    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-serving-cert\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.175433 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.175415    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-oauth-config\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.180537 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.180512    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgp59\" (UniqueName: \"kubernetes.io/projected/e57ce052-bf4e-4667-a732-7601bdee84bf-kube-api-access-fgp59\") pod \"console-758c75c9c6-2468d\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") " pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.244238 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.244201    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:15.366714 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.366624    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758c75c9c6-2468d"]
May 29 02:01:15.370615 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:01:15.370573    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57ce052_bf4e_4667_a732_7601bdee84bf.slice/crio-cb4d43643739a2dd23322738832af38ea4a7304e883d1848ccb07a78ccf4f7f0 WatchSource:0}: Error finding container cb4d43643739a2dd23322738832af38ea4a7304e883d1848ccb07a78ccf4f7f0: Status 404 returned error can't find the container with id cb4d43643739a2dd23322738832af38ea4a7304e883d1848ccb07a78ccf4f7f0
May 29 02:01:15.511149 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.511112    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758c75c9c6-2468d" event={"ID":"e57ce052-bf4e-4667-a732-7601bdee84bf","Type":"ContainerStarted","Data":"cb4d43643739a2dd23322738832af38ea4a7304e883d1848ccb07a78ccf4f7f0"}
May 29 02:01:15.514465 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:15.514415    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fqm6l" event={"ID":"2cd392d5-8760-45f1-9164-4d9c714a6de3","Type":"ContainerStarted","Data":"87d9eb0854259312390bf0c9ad03bb7cf9690e9d1b583d9ecca7e08484ff5ae5"}
May 29 02:01:16.518218 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:16.518180    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758c75c9c6-2468d" event={"ID":"e57ce052-bf4e-4667-a732-7601bdee84bf","Type":"ContainerStarted","Data":"3cb5b0f7677477cb0d2f19fb8d3cd1468d4ea1d55dae7c4c51a8c324bad7f79f"}
May 29 02:01:16.541647 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:16.541444    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-758c75c9c6-2468d" podStartSLOduration=2.541424011 podStartE2EDuration="2.541424011s" podCreationTimestamp="2026-05-29 02:01:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:01:16.540949209 +0000 UTC m=+173.123940113" watchObservedRunningTime="2026-05-29 02:01:16.541424011 +0000 UTC m=+173.124414916"
May 29 02:01:17.522809 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:17.522720    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fqm6l" event={"ID":"2cd392d5-8760-45f1-9164-4d9c714a6de3","Type":"ContainerStarted","Data":"abc05ef9d75d981f9e0515125bab44acd315730a2228a1ae5dbfb31fb6900338"}
May 29 02:01:17.522809 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:17.522760    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fqm6l" event={"ID":"2cd392d5-8760-45f1-9164-4d9c714a6de3","Type":"ContainerStarted","Data":"35e6bcfb8bcd1dd57831a41400a2f63772dec80efd760710ec8ce910b8f4731f"}
May 29 02:01:17.523212 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:17.522815    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fqm6l"
May 29 02:01:17.542224 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:17.542173    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-fqm6l" podStartSLOduration=138.690543144 podStartE2EDuration="2m20.542158176s" podCreationTimestamp="2026-05-29 01:58:57 +0000 UTC" firstStartedPulling="2026-05-29 02:01:15.113115231 +0000 UTC m=+171.696106115" lastFinishedPulling="2026-05-29 02:01:16.964730248 +0000 UTC m=+173.547721147" observedRunningTime="2026-05-29 02:01:17.540962204 +0000 UTC m=+174.123953110" watchObservedRunningTime="2026-05-29 02:01:17.542158176 +0000 UTC m=+174.125149080"
May 29 02:01:20.621688 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:20.621655    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:23.543606 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:23.543565    2580 generic.go:358] "Generic (PLEG): container finished" podID="37947119-0606-40c1-9ea3-eb3005ce71e3" containerID="1f371ddb8a4169d32ff75884a341dd9fad66ac67375fde3b82cad4c78d53b936" exitCode=0
May 29 02:01:23.543971 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:23.543645    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph" event={"ID":"37947119-0606-40c1-9ea3-eb3005ce71e3","Type":"ContainerDied","Data":"1f371ddb8a4169d32ff75884a341dd9fad66ac67375fde3b82cad4c78d53b936"}
May 29 02:01:23.544011 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:23.543971    2580 scope.go:117] "RemoveContainer" containerID="1f371ddb8a4169d32ff75884a341dd9fad66ac67375fde3b82cad4c78d53b936"
May 29 02:01:24.552391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:24.552353    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-h7sph" event={"ID":"37947119-0606-40c1-9ea3-eb3005ce71e3","Type":"ContainerStarted","Data":"3312d3499b4f541106caf3431ec13ca7bc44c5ff9a18f7d280059234595931f8"}
May 29 02:01:25.244890 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:25.244854    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:25.245048 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:25.244902    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:25.249564 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:25.249542    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:25.558992 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:25.558959    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:01:27.527896 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:27.527866    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fqm6l"
May 29 02:01:29.006321 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:29.006286    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:29.006321 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:29.006325    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:35.163442 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.163380    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-67c96447c9-cfktq" podUID="adc062cf-dc64-4df1-b444-d01cb21b4e47" containerName="console" containerID="cri-o://c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd" gracePeriod=15
May 29 02:01:35.419776 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.419718    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67c96447c9-cfktq_adc062cf-dc64-4df1-b444-d01cb21b4e47/console/0.log"
May 29 02:01:35.419878 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.419803    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:01:35.437355 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.437332    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-oauth-config\") pod \"adc062cf-dc64-4df1-b444-d01cb21b4e47\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") "
May 29 02:01:35.437501 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.437362    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw78j\" (UniqueName: \"kubernetes.io/projected/adc062cf-dc64-4df1-b444-d01cb21b4e47-kube-api-access-jw78j\") pod \"adc062cf-dc64-4df1-b444-d01cb21b4e47\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") "
May 29 02:01:35.437501 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.437424    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-service-ca\") pod \"adc062cf-dc64-4df1-b444-d01cb21b4e47\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") "
May 29 02:01:35.437501 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.437459    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-oauth-serving-cert\") pod \"adc062cf-dc64-4df1-b444-d01cb21b4e47\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") "
May 29 02:01:35.437501 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.437498    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-serving-cert\") pod \"adc062cf-dc64-4df1-b444-d01cb21b4e47\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") "
May 29 02:01:35.437737 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.437538    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-config\") pod \"adc062cf-dc64-4df1-b444-d01cb21b4e47\" (UID: \"adc062cf-dc64-4df1-b444-d01cb21b4e47\") "
May 29 02:01:35.438517 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.438340    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-config" (OuterVolumeSpecName: "console-config") pod "adc062cf-dc64-4df1-b444-d01cb21b4e47" (UID: "adc062cf-dc64-4df1-b444-d01cb21b4e47"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:01:35.438517 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.438354    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "adc062cf-dc64-4df1-b444-d01cb21b4e47" (UID: "adc062cf-dc64-4df1-b444-d01cb21b4e47"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:01:35.438517 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.438337    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-service-ca" (OuterVolumeSpecName: "service-ca") pod "adc062cf-dc64-4df1-b444-d01cb21b4e47" (UID: "adc062cf-dc64-4df1-b444-d01cb21b4e47"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:01:35.441134 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.441095    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "adc062cf-dc64-4df1-b444-d01cb21b4e47" (UID: "adc062cf-dc64-4df1-b444-d01cb21b4e47"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:01:35.441272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.441113    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "adc062cf-dc64-4df1-b444-d01cb21b4e47" (UID: "adc062cf-dc64-4df1-b444-d01cb21b4e47"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:01:35.441272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.441139    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc062cf-dc64-4df1-b444-d01cb21b4e47-kube-api-access-jw78j" (OuterVolumeSpecName: "kube-api-access-jw78j") pod "adc062cf-dc64-4df1-b444-d01cb21b4e47" (UID: "adc062cf-dc64-4df1-b444-d01cb21b4e47"). InnerVolumeSpecName "kube-api-access-jw78j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:01:35.538925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.538888    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-oauth-serving-cert\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:35.538925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.538918    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-serving-cert\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:35.538925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.538928    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-config\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:35.539152 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.538937    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adc062cf-dc64-4df1-b444-d01cb21b4e47-console-oauth-config\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:35.539152 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.538947    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw78j\" (UniqueName: \"kubernetes.io/projected/adc062cf-dc64-4df1-b444-d01cb21b4e47-kube-api-access-jw78j\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:35.539152 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.538956    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adc062cf-dc64-4df1-b444-d01cb21b4e47-service-ca\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:35.585729 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.585703    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67c96447c9-cfktq_adc062cf-dc64-4df1-b444-d01cb21b4e47/console/0.log"
May 29 02:01:35.585908 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.585743    2580 generic.go:358] "Generic (PLEG): container finished" podID="adc062cf-dc64-4df1-b444-d01cb21b4e47" containerID="c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd" exitCode=2
May 29 02:01:35.585908 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.585779    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c96447c9-cfktq" event={"ID":"adc062cf-dc64-4df1-b444-d01cb21b4e47","Type":"ContainerDied","Data":"c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd"}
May 29 02:01:35.585908 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.585816    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c96447c9-cfktq"
May 29 02:01:35.585908 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.585828    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c96447c9-cfktq" event={"ID":"adc062cf-dc64-4df1-b444-d01cb21b4e47","Type":"ContainerDied","Data":"348759d4c0f5e4e7281fb31a827ce5e9a5b4adf3e0a753a4ad812740f711fd86"}
May 29 02:01:35.585908 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.585849    2580 scope.go:117] "RemoveContainer" containerID="c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd"
May 29 02:01:35.594321 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.594302    2580 scope.go:117] "RemoveContainer" containerID="c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd"
May 29 02:01:35.594623 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:01:35.594586    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd\": container with ID starting with c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd not found: ID does not exist" containerID="c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd"
May 29 02:01:35.594693 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.594632    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd"} err="failed to get container status \"c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd\": rpc error: code = NotFound desc = could not find container \"c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd\": container with ID starting with c4b9a0f8c7eb9476632e9841df1428f9d2c057bdbd8397ece97e0ab3f7a2c3fd not found: ID does not exist"
May 29 02:01:35.608393 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.608370    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c96447c9-cfktq"]
May 29 02:01:35.611443 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.611423    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67c96447c9-cfktq"]
May 29 02:01:35.974308 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:35.974276    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adc062cf-dc64-4df1-b444-d01cb21b4e47" path="/var/lib/kubelet/pods/adc062cf-dc64-4df1-b444-d01cb21b4e47/volumes"
May 29 02:01:39.884208 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:39.884152    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-84cddbdcf5-hfwq8" podUID="c537270a-1d71-431e-ace3-385999191fa4" containerName="console" containerID="cri-o://bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0" gracePeriod=15
May 29 02:01:40.133621 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.133569    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84cddbdcf5-hfwq8_c537270a-1d71-431e-ace3-385999191fa4/console/0.log"
May 29 02:01:40.133734 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.133671    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:40.178074 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.177996    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-oauth-serving-cert\") pod \"c537270a-1d71-431e-ace3-385999191fa4\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") "
May 29 02:01:40.178074 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178036    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-trusted-ca-bundle\") pod \"c537270a-1d71-431e-ace3-385999191fa4\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") "
May 29 02:01:40.178074 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178059    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pthw\" (UniqueName: \"kubernetes.io/projected/c537270a-1d71-431e-ace3-385999191fa4-kube-api-access-4pthw\") pod \"c537270a-1d71-431e-ace3-385999191fa4\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") "
May 29 02:01:40.178344 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178089    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-oauth-config\") pod \"c537270a-1d71-431e-ace3-385999191fa4\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") "
May 29 02:01:40.178344 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178140    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-console-config\") pod \"c537270a-1d71-431e-ace3-385999191fa4\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") "
May 29 02:01:40.178344 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178165    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-service-ca\") pod \"c537270a-1d71-431e-ace3-385999191fa4\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") "
May 29 02:01:40.178344 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178197    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-serving-cert\") pod \"c537270a-1d71-431e-ace3-385999191fa4\" (UID: \"c537270a-1d71-431e-ace3-385999191fa4\") "
May 29 02:01:40.178640 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178615    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c537270a-1d71-431e-ace3-385999191fa4" (UID: "c537270a-1d71-431e-ace3-385999191fa4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:01:40.178709 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178631    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c537270a-1d71-431e-ace3-385999191fa4" (UID: "c537270a-1d71-431e-ace3-385999191fa4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:01:40.178709 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178646    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-console-config" (OuterVolumeSpecName: "console-config") pod "c537270a-1d71-431e-ace3-385999191fa4" (UID: "c537270a-1d71-431e-ace3-385999191fa4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:01:40.178813 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.178761    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-service-ca" (OuterVolumeSpecName: "service-ca") pod "c537270a-1d71-431e-ace3-385999191fa4" (UID: "c537270a-1d71-431e-ace3-385999191fa4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:01:40.180317 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.180294    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c537270a-1d71-431e-ace3-385999191fa4" (UID: "c537270a-1d71-431e-ace3-385999191fa4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:01:40.180317 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.180300    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c537270a-1d71-431e-ace3-385999191fa4-kube-api-access-4pthw" (OuterVolumeSpecName: "kube-api-access-4pthw") pod "c537270a-1d71-431e-ace3-385999191fa4" (UID: "c537270a-1d71-431e-ace3-385999191fa4"). InnerVolumeSpecName "kube-api-access-4pthw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:01:40.180458 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.180321    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c537270a-1d71-431e-ace3-385999191fa4" (UID: "c537270a-1d71-431e-ace3-385999191fa4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:01:40.278978 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.278946    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-serving-cert\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:40.278978 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.278975    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-oauth-serving-cert\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:40.278978 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.278985    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-trusted-ca-bundle\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:40.279199 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.278996    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pthw\" (UniqueName: \"kubernetes.io/projected/c537270a-1d71-431e-ace3-385999191fa4-kube-api-access-4pthw\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:40.279199 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.279005    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c537270a-1d71-431e-ace3-385999191fa4-console-oauth-config\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:40.279199 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.279014    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-console-config\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:40.279199 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.279023    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c537270a-1d71-431e-ace3-385999191fa4-service-ca\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:01:40.601864 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.601833    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84cddbdcf5-hfwq8_c537270a-1d71-431e-ace3-385999191fa4/console/0.log"
May 29 02:01:40.602044 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.601883    2580 generic.go:358] "Generic (PLEG): container finished" podID="c537270a-1d71-431e-ace3-385999191fa4" containerID="bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0" exitCode=2
May 29 02:01:40.602044 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.601950    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cddbdcf5-hfwq8"
May 29 02:01:40.602044 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.601973    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cddbdcf5-hfwq8" event={"ID":"c537270a-1d71-431e-ace3-385999191fa4","Type":"ContainerDied","Data":"bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0"}
May 29 02:01:40.602044 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.602018    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cddbdcf5-hfwq8" event={"ID":"c537270a-1d71-431e-ace3-385999191fa4","Type":"ContainerDied","Data":"3ea9bcb1f046111cd283fa3ed148e9c40fbbbde9acea74ae94dcea09fd823493"}
May 29 02:01:40.602044 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.602038    2580 scope.go:117] "RemoveContainer" containerID="bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0"
May 29 02:01:40.609973 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.609955    2580 scope.go:117] "RemoveContainer" containerID="bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0"
May 29 02:01:40.610231 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:01:40.610211    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0\": container with ID starting with bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0 not found: ID does not exist" containerID="bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0"
May 29 02:01:40.610306 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.610238    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0"} err="failed to get container status \"bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0\": rpc error: code = NotFound desc = could not find container \"bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0\": container with ID starting with bf63781b1d3104d95913258f337cb6b16361b9ffef961aaf7cb9cbf2796c6ef0 not found: ID does not exist"
May 29 02:01:40.623154 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.623130    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84cddbdcf5-hfwq8"]
May 29 02:01:40.628106 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:40.628084    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84cddbdcf5-hfwq8"]
May 29 02:01:41.974217 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:41.974184    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c537270a-1d71-431e-ace3-385999191fa4" path="/var/lib/kubelet/pods/c537270a-1d71-431e-ace3-385999191fa4/volumes"
May 29 02:01:43.611668 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:43.611630    2580 generic.go:358] "Generic (PLEG): container finished" podID="eedde3d0-971e-40bf-97f8-5c21637f916f" containerID="75e0387cbafd70ef8ac093d285660c629a296bdba1c5a415ba345798ef89b457" exitCode=0
May 29 02:01:43.612030 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:43.611701    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld" event={"ID":"eedde3d0-971e-40bf-97f8-5c21637f916f","Type":"ContainerDied","Data":"75e0387cbafd70ef8ac093d285660c629a296bdba1c5a415ba345798ef89b457"}
May 29 02:01:43.612030 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:43.612012    2580 scope.go:117] "RemoveContainer" containerID="75e0387cbafd70ef8ac093d285660c629a296bdba1c5a415ba345798ef89b457"
May 29 02:01:44.616258 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:44.616223    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-t6jld" event={"ID":"eedde3d0-971e-40bf-97f8-5c21637f916f","Type":"ContainerStarted","Data":"daa20359216ae337e95205cb1c1524f69f29144ed47fa42b6b6480967b866ddb"}
May 29 02:01:49.011577 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:49.011545    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:01:49.015278 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:01:49.015256    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bb8bb4fc7-2n67j"
May 29 02:02:26.523610 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.523560    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b665c8654-4vm8b"]
May 29 02:02:26.524059 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.523976    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adc062cf-dc64-4df1-b444-d01cb21b4e47" containerName="console"
May 29 02:02:26.524059 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.523993    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adc062cf-dc64-4df1-b444-d01cb21b4e47" containerName="console"
May 29 02:02:26.524059 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.524009    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c537270a-1d71-431e-ace3-385999191fa4" containerName="console"
May 29 02:02:26.524059 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.524017    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c537270a-1d71-431e-ace3-385999191fa4" containerName="console"
May 29 02:02:26.524211 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.524074    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="c537270a-1d71-431e-ace3-385999191fa4" containerName="console"
May 29 02:02:26.524211 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.524081    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="adc062cf-dc64-4df1-b444-d01cb21b4e47" containerName="console"
May 29 02:02:26.528308 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.528289    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.551290 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.551269    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b665c8654-4vm8b"]
May 29 02:02:26.647212 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.647175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-oauth-serving-cert\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.647372 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.647234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-trusted-ca-bundle\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.647372 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.647304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9d9\" (UniqueName: \"kubernetes.io/projected/0840238b-c27f-4e12-9d64-85ea466572af-kube-api-access-lt9d9\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.647372 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.647344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-oauth-config\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.647473 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.647376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-serving-cert\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.647473 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.647391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-console-config\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.647473 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.647451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-service-ca\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.748186 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.748157    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-service-ca\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.748346 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.748213    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-oauth-serving-cert\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.748346 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.748257    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-trusted-ca-bundle\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.748346 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.748283    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9d9\" (UniqueName: \"kubernetes.io/projected/0840238b-c27f-4e12-9d64-85ea466572af-kube-api-access-lt9d9\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.748346 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.748314    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-oauth-config\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.748346 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.748346    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-serving-cert\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.748624 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.748368    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-console-config\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.749075 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.749054    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-oauth-serving-cert\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.749075 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.749065    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-service-ca\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.749209 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.749068    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-console-config\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.749361 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.749337    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-trusted-ca-bundle\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.750921 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.750903    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-oauth-config\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.751012 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.750996    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-serving-cert\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.765991 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.765972    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9d9\" (UniqueName: \"kubernetes.io/projected/0840238b-c27f-4e12-9d64-85ea466572af-kube-api-access-lt9d9\") pod \"console-5b665c8654-4vm8b\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") " pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.837073 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.837043    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:26.975240 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:26.975174    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b665c8654-4vm8b"]
May 29 02:02:26.978894 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:02:26.978866    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0840238b_c27f_4e12_9d64_85ea466572af.slice/crio-ca8176f0e66ea13160f543fbdd17234ce72a53e151091cfa9e8602f4b46742bc WatchSource:0}: Error finding container ca8176f0e66ea13160f543fbdd17234ce72a53e151091cfa9e8602f4b46742bc: Status 404 returned error can't find the container with id ca8176f0e66ea13160f543fbdd17234ce72a53e151091cfa9e8602f4b46742bc
May 29 02:02:27.744473 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:27.744424    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b665c8654-4vm8b" event={"ID":"0840238b-c27f-4e12-9d64-85ea466572af","Type":"ContainerStarted","Data":"47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f"}
May 29 02:02:27.744473 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:27.744476    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b665c8654-4vm8b" event={"ID":"0840238b-c27f-4e12-9d64-85ea466572af","Type":"ContainerStarted","Data":"ca8176f0e66ea13160f543fbdd17234ce72a53e151091cfa9e8602f4b46742bc"}
May 29 02:02:27.771954 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:27.771906    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5b665c8654-4vm8b" podStartSLOduration=1.771892456 podStartE2EDuration="1.771892456s" podCreationTimestamp="2026-05-29 02:02:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:02:27.771390288 +0000 UTC m=+244.354381193" watchObservedRunningTime="2026-05-29 02:02:27.771892456 +0000 UTC m=+244.354883361"
May 29 02:02:36.838166 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:36.838124    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:36.838166 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:36.838172    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:36.842780 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:36.842754    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:37.777077 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:37.777047    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:02:37.897025 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:37.896988    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758c75c9c6-2468d"]
May 29 02:02:59.319425 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.319384    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vb52g"]
May 29 02:02:59.323075 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.323051    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.326428 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.326410    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 29 02:02:59.342182 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.342154    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vb52g"]
May 29 02:02:59.386685 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.386654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6c5c410-be0e-4f48-9ace-04f0312622a5-kubelet-config\") pod \"global-pull-secret-syncer-vb52g\" (UID: \"c6c5c410-be0e-4f48-9ace-04f0312622a5\") " pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.386838 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.386753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6c5c410-be0e-4f48-9ace-04f0312622a5-original-pull-secret\") pod \"global-pull-secret-syncer-vb52g\" (UID: \"c6c5c410-be0e-4f48-9ace-04f0312622a5\") " pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.386838 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.386779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6c5c410-be0e-4f48-9ace-04f0312622a5-dbus\") pod \"global-pull-secret-syncer-vb52g\" (UID: \"c6c5c410-be0e-4f48-9ace-04f0312622a5\") " pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.487734 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.487700    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6c5c410-be0e-4f48-9ace-04f0312622a5-original-pull-secret\") pod \"global-pull-secret-syncer-vb52g\" (UID: \"c6c5c410-be0e-4f48-9ace-04f0312622a5\") " pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.487734 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.487734    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6c5c410-be0e-4f48-9ace-04f0312622a5-dbus\") pod \"global-pull-secret-syncer-vb52g\" (UID: \"c6c5c410-be0e-4f48-9ace-04f0312622a5\") " pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.487899 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.487764    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6c5c410-be0e-4f48-9ace-04f0312622a5-kubelet-config\") pod \"global-pull-secret-syncer-vb52g\" (UID: \"c6c5c410-be0e-4f48-9ace-04f0312622a5\") " pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.487899 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.487872    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6c5c410-be0e-4f48-9ace-04f0312622a5-kubelet-config\") pod \"global-pull-secret-syncer-vb52g\" (UID: \"c6c5c410-be0e-4f48-9ace-04f0312622a5\") " pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.487962 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.487919    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6c5c410-be0e-4f48-9ace-04f0312622a5-dbus\") pod \"global-pull-secret-syncer-vb52g\" (UID: \"c6c5c410-be0e-4f48-9ace-04f0312622a5\") " pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.490057 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.490024    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6c5c410-be0e-4f48-9ace-04f0312622a5-original-pull-secret\") pod \"global-pull-secret-syncer-vb52g\" (UID: \"c6c5c410-be0e-4f48-9ace-04f0312622a5\") " pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.632509 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.632407    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb52g"
May 29 02:02:59.770239 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.770076    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vb52g"]
May 29 02:02:59.772914 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:02:59.772885    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c5c410_be0e_4f48_9ace_04f0312622a5.slice/crio-5836baf34ffff78ba105cb04e92c1ac01c879745169a5e36d7e3fc1d8fb8b85a WatchSource:0}: Error finding container 5836baf34ffff78ba105cb04e92c1ac01c879745169a5e36d7e3fc1d8fb8b85a: Status 404 returned error can't find the container with id 5836baf34ffff78ba105cb04e92c1ac01c879745169a5e36d7e3fc1d8fb8b85a
May 29 02:02:59.834073 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:02:59.834036    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vb52g" event={"ID":"c6c5c410-be0e-4f48-9ace-04f0312622a5","Type":"ContainerStarted","Data":"5836baf34ffff78ba105cb04e92c1ac01c879745169a5e36d7e3fc1d8fb8b85a"}
May 29 02:03:02.915405 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:02.915359    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-758c75c9c6-2468d" podUID="e57ce052-bf4e-4667-a732-7601bdee84bf" containerName="console" containerID="cri-o://3cb5b0f7677477cb0d2f19fb8d3cd1468d4ea1d55dae7c4c51a8c324bad7f79f" gracePeriod=15
May 29 02:03:03.847572 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:03.847541    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-758c75c9c6-2468d_e57ce052-bf4e-4667-a732-7601bdee84bf/console/0.log"
May 29 02:03:03.847736 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:03.847587    2580 generic.go:358] "Generic (PLEG): container finished" podID="e57ce052-bf4e-4667-a732-7601bdee84bf" containerID="3cb5b0f7677477cb0d2f19fb8d3cd1468d4ea1d55dae7c4c51a8c324bad7f79f" exitCode=2
May 29 02:03:03.847736 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:03.847626    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758c75c9c6-2468d" event={"ID":"e57ce052-bf4e-4667-a732-7601bdee84bf","Type":"ContainerDied","Data":"3cb5b0f7677477cb0d2f19fb8d3cd1468d4ea1d55dae7c4c51a8c324bad7f79f"}
May 29 02:03:04.476143 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.476122    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-758c75c9c6-2468d_e57ce052-bf4e-4667-a732-7601bdee84bf/console/0.log"
May 29 02:03:04.476441 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.476188    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:03:04.530505 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.530481    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-trusted-ca-bundle\") pod \"e57ce052-bf4e-4667-a732-7601bdee84bf\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") "
May 29 02:03:04.530623 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.530543    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-oauth-config\") pod \"e57ce052-bf4e-4667-a732-7601bdee84bf\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") "
May 29 02:03:04.530623 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.530572    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgp59\" (UniqueName: \"kubernetes.io/projected/e57ce052-bf4e-4667-a732-7601bdee84bf-kube-api-access-fgp59\") pod \"e57ce052-bf4e-4667-a732-7601bdee84bf\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") "
May 29 02:03:04.530739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.530621    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-service-ca\") pod \"e57ce052-bf4e-4667-a732-7601bdee84bf\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") "
May 29 02:03:04.530739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.530647    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-oauth-serving-cert\") pod \"e57ce052-bf4e-4667-a732-7601bdee84bf\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") "
May 29 02:03:04.530739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.530682    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-serving-cert\") pod \"e57ce052-bf4e-4667-a732-7601bdee84bf\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") "
May 29 02:03:04.530739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.530719    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-console-config\") pod \"e57ce052-bf4e-4667-a732-7601bdee84bf\" (UID: \"e57ce052-bf4e-4667-a732-7601bdee84bf\") "
May 29 02:03:04.530934 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.530834    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e57ce052-bf4e-4667-a732-7601bdee84bf" (UID: "e57ce052-bf4e-4667-a732-7601bdee84bf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:03:04.531277 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.531129    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-console-config" (OuterVolumeSpecName: "console-config") pod "e57ce052-bf4e-4667-a732-7601bdee84bf" (UID: "e57ce052-bf4e-4667-a732-7601bdee84bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:03:04.531277 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.531213    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e57ce052-bf4e-4667-a732-7601bdee84bf" (UID: "e57ce052-bf4e-4667-a732-7601bdee84bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:03:04.531497 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.531476    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-service-ca" (OuterVolumeSpecName: "service-ca") pod "e57ce052-bf4e-4667-a732-7601bdee84bf" (UID: "e57ce052-bf4e-4667-a732-7601bdee84bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:03:04.538637 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.533327    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57ce052-bf4e-4667-a732-7601bdee84bf-kube-api-access-fgp59" (OuterVolumeSpecName: "kube-api-access-fgp59") pod "e57ce052-bf4e-4667-a732-7601bdee84bf" (UID: "e57ce052-bf4e-4667-a732-7601bdee84bf"). InnerVolumeSpecName "kube-api-access-fgp59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:03:04.539484 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.539386    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e57ce052-bf4e-4667-a732-7601bdee84bf" (UID: "e57ce052-bf4e-4667-a732-7601bdee84bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:03:04.539484 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.539455    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e57ce052-bf4e-4667-a732-7601bdee84bf" (UID: "e57ce052-bf4e-4667-a732-7601bdee84bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:03:04.631323 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.631293    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-service-ca\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:03:04.631323 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.631319    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-oauth-serving-cert\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:03:04.631323 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.631329    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-serving-cert\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:03:04.631558 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.631338    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-console-config\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:03:04.631558 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.631347    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57ce052-bf4e-4667-a732-7601bdee84bf-trusted-ca-bundle\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:03:04.631558 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.631356    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e57ce052-bf4e-4667-a732-7601bdee84bf-console-oauth-config\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:03:04.631558 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.631364    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgp59\" (UniqueName: \"kubernetes.io/projected/e57ce052-bf4e-4667-a732-7601bdee84bf-kube-api-access-fgp59\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:03:04.851304 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.851280    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-758c75c9c6-2468d_e57ce052-bf4e-4667-a732-7601bdee84bf/console/0.log"
May 29 02:03:04.851450 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.851380    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758c75c9c6-2468d" event={"ID":"e57ce052-bf4e-4667-a732-7601bdee84bf","Type":"ContainerDied","Data":"cb4d43643739a2dd23322738832af38ea4a7304e883d1848ccb07a78ccf4f7f0"}
May 29 02:03:04.851450 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.851421    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758c75c9c6-2468d"
May 29 02:03:04.851555 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.851424    2580 scope.go:117] "RemoveContainer" containerID="3cb5b0f7677477cb0d2f19fb8d3cd1468d4ea1d55dae7c4c51a8c324bad7f79f"
May 29 02:03:04.852807 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.852783    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vb52g" event={"ID":"c6c5c410-be0e-4f48-9ace-04f0312622a5","Type":"ContainerStarted","Data":"1126cd82c93c84abbb3be9746495b2981a54380949889a92699484ec95ff2cfd"}
May 29 02:03:04.876941 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.876898    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vb52g" podStartSLOduration=1.133562791 podStartE2EDuration="5.876885528s" podCreationTimestamp="2026-05-29 02:02:59 +0000 UTC" firstStartedPulling="2026-05-29 02:02:59.774571763 +0000 UTC m=+276.357562652" lastFinishedPulling="2026-05-29 02:03:04.517894503 +0000 UTC m=+281.100885389" observedRunningTime="2026-05-29 02:03:04.876828937 +0000 UTC m=+281.459819844" watchObservedRunningTime="2026-05-29 02:03:04.876885528 +0000 UTC m=+281.459876432"
May 29 02:03:04.899689 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.899657    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758c75c9c6-2468d"]
May 29 02:03:04.906732 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:04.906706    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-758c75c9c6-2468d"]
May 29 02:03:05.974510 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:05.974481    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e57ce052-bf4e-4667-a732-7601bdee84bf" path="/var/lib/kubelet/pods/e57ce052-bf4e-4667-a732-7601bdee84bf/volumes"
May 29 02:03:23.848205 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:23.848178    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:03:23.848700 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:23.848179    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:03:23.851332 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:23.851312    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:03:23.851466 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:23.851318    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:03:23.862188 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:03:23.862166    2580 kubelet.go:1628] "Image garbage collection succeeded"
May 29 02:04:19.953964 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.953886    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"]
May 29 02:04:19.954339 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.954204    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e57ce052-bf4e-4667-a732-7601bdee84bf" containerName="console"
May 29 02:04:19.954339 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.954216    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e57ce052-bf4e-4667-a732-7601bdee84bf" containerName="console"
May 29 02:04:19.954339 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.954271    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="e57ce052-bf4e-4667-a732-7601bdee84bf" containerName="console"
May 29 02:04:19.957088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.957071    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"
May 29 02:04:19.959211 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.959187    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 29 02:04:19.960232 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.960207    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 29 02:04:19.960326 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.960288    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-zf8zh\""
May 29 02:04:19.966953 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.966924    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"]
May 29 02:04:19.977956 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:19.977927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86bs\" (UniqueName: \"kubernetes.io/projected/f6bf796f-3929-41e1-bfed-b56adec38559-kube-api-access-c86bs\") pod \"feast-operator-controller-manager-7b4c9ffb8c-fp49m\" (UID: \"f6bf796f-3929-41e1-bfed-b56adec38559\") " pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"
May 29 02:04:20.078588 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:20.078549    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c86bs\" (UniqueName: \"kubernetes.io/projected/f6bf796f-3929-41e1-bfed-b56adec38559-kube-api-access-c86bs\") pod \"feast-operator-controller-manager-7b4c9ffb8c-fp49m\" (UID: \"f6bf796f-3929-41e1-bfed-b56adec38559\") " pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"
May 29 02:04:20.086293 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:20.086271    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86bs\" (UniqueName: \"kubernetes.io/projected/f6bf796f-3929-41e1-bfed-b56adec38559-kube-api-access-c86bs\") pod \"feast-operator-controller-manager-7b4c9ffb8c-fp49m\" (UID: \"f6bf796f-3929-41e1-bfed-b56adec38559\") " pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"
May 29 02:04:20.272179 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:20.272094    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"
May 29 02:04:20.397390 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:20.397359    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"]
May 29 02:04:20.400411 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:04:20.400380    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bf796f_3929_41e1_bfed_b56adec38559.slice/crio-f39f2b0b524e7dcd404e3061a3d4749ba7c599f661e0a55e6d35bb5e58c36c43 WatchSource:0}: Error finding container f39f2b0b524e7dcd404e3061a3d4749ba7c599f661e0a55e6d35bb5e58c36c43: Status 404 returned error can't find the container with id f39f2b0b524e7dcd404e3061a3d4749ba7c599f661e0a55e6d35bb5e58c36c43
May 29 02:04:20.402231 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:20.402214    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 02:04:21.062054 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:21.062011    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m" event={"ID":"f6bf796f-3929-41e1-bfed-b56adec38559","Type":"ContainerStarted","Data":"f39f2b0b524e7dcd404e3061a3d4749ba7c599f661e0a55e6d35bb5e58c36c43"}
May 29 02:04:23.068690 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:23.068655    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m" event={"ID":"f6bf796f-3929-41e1-bfed-b56adec38559","Type":"ContainerStarted","Data":"776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854"}
May 29 02:04:23.069073 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:23.068767    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"
May 29 02:04:23.083860 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:23.083810    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m" podStartSLOduration=1.801463158 podStartE2EDuration="4.083795786s" podCreationTimestamp="2026-05-29 02:04:19 +0000 UTC" firstStartedPulling="2026-05-29 02:04:20.40236472 +0000 UTC m=+356.985355603" lastFinishedPulling="2026-05-29 02:04:22.684697347 +0000 UTC m=+359.267688231" observedRunningTime="2026-05-29 02:04:23.083024083 +0000 UTC m=+359.666014990" watchObservedRunningTime="2026-05-29 02:04:23.083795786 +0000 UTC m=+359.666786738"
May 29 02:04:34.074081 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:04:34.074046    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"
May 29 02:05:01.580381 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.580350    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"]
May 29 02:05:01.587486 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.587468    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.589712 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.589687    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 29 02:05:01.589840 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.589753    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-jw8v5\""
May 29 02:05:01.589840 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.589805    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 02:05:01.589840 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.589819    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 02:05:01.590009 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.589879    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 29 02:05:01.590009 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.589881    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 29 02:05:01.590009 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.589937    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 29 02:05:01.590198 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.590184    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 29 02:05:01.596492 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.596471    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"]
May 29 02:05:01.682937 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.682906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-online-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.682937 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.682938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-feast-data\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.683163 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.683004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glvjm\" (UniqueName: \"kubernetes.io/projected/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-kube-api-access-glvjm\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.683163 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.683030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-registry-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.683163 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.683114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-offline-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.683297 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.683166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-ui-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.784131 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.784095    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-ui-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.784317 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.784151    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-online-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.784317 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.784170    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-feast-data\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.784317 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.784203    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-glvjm\" (UniqueName: \"kubernetes.io/projected/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-kube-api-access-glvjm\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.784317 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.784223    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-registry-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.784317 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.784247    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-offline-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.784317 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:05:01.784254    2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 29 02:05:01.784621 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:05:01.784336    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-ui-tls podName:dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9 nodeName:}" failed. No retries permitted until 2026-05-29 02:05:02.284315348 +0000 UTC m=+398.867306247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-ui-tls") pod "feast-simple-feast-setup-675665f868-sczcv" (UID: "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9") : secret "feast-simple-feast-setup-ui-tls" not found
May 29 02:05:01.784621 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:05:01.784351    2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 29 02:05:01.784621 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:05:01.784426    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-registry-tls podName:dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9 nodeName:}" failed. No retries permitted until 2026-05-29 02:05:02.284407356 +0000 UTC m=+398.867398258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-registry-tls") pod "feast-simple-feast-setup-675665f868-sczcv" (UID: "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9") : secret "feast-simple-feast-setup-registry-tls" not found
May 29 02:05:01.784739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.784675    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-feast-data\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.786653 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.786629    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-offline-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.786747 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.786681    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-online-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:01.795414 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:01.795391    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glvjm\" (UniqueName: \"kubernetes.io/projected/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-kube-api-access-glvjm\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:02.286834 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:02.286790    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-ui-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:02.286999 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:02.286877    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-registry-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:02.289265 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:02.289235    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-ui-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:02.289265 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:02.289254    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-registry-tls\") pod \"feast-simple-feast-setup-675665f868-sczcv\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:02.505990 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:02.505948    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:02.644067 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:02.644037    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"]
May 29 02:05:02.644674 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:05:02.644651    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbf379ab_bfa4_4350_84cf_cbd0c63eb7d9.slice/crio-b9d54ed28cf5c47213d12cef7308e4d0db43431faa58246365e71ab2a25feb4a WatchSource:0}: Error finding container b9d54ed28cf5c47213d12cef7308e4d0db43431faa58246365e71ab2a25feb4a: Status 404 returned error can't find the container with id b9d54ed28cf5c47213d12cef7308e4d0db43431faa58246365e71ab2a25feb4a
May 29 02:05:03.181656 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:03.181616    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerStarted","Data":"b9d54ed28cf5c47213d12cef7308e4d0db43431faa58246365e71ab2a25feb4a"}
May 29 02:05:36.283724 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:36.283687    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerStarted","Data":"1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410"}
May 29 02:05:40.298934 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:40.298899    2580 generic.go:358] "Generic (PLEG): container finished" podID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerID="1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410" exitCode=0
May 29 02:05:40.299233 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:40.298971    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerDied","Data":"1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410"}
May 29 02:05:41.304604 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:41.304552    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerStarted","Data":"836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078"}
May 29 02:05:46.320973 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:46.320939    2580 generic.go:358] "Generic (PLEG): container finished" podID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerID="836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078" exitCode=0
May 29 02:05:46.320973 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:46.320979    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerDied","Data":"836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078"}
May 29 02:05:46.330800 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:46.330778    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 29 02:05:47.328298 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.328260    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerStarted","Data":"493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba"}
May 29 02:05:47.328298 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.328305    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerStarted","Data":"d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a"}
May 29 02:05:47.328836 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.328322    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerStarted","Data":"600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561"}
May 29 02:05:47.328836 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.328337    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerStarted","Data":"1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a"}
May 29 02:05:47.352900 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.352758    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podStartSLOduration=13.308890702 podStartE2EDuration="46.352737059s" podCreationTimestamp="2026-05-29 02:05:01 +0000 UTC" firstStartedPulling="2026-05-29 02:05:02.646427877 +0000 UTC m=+399.229418759" lastFinishedPulling="2026-05-29 02:05:35.690274233 +0000 UTC m=+432.273265116" observedRunningTime="2026-05-29 02:05:47.351908643 +0000 UTC m=+443.934899553" watchObservedRunningTime="2026-05-29 02:05:47.352737059 +0000 UTC m=+443.935727965"
May 29 02:05:47.506911 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.506795    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:47.506911 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.506849    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:47.506911 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.506865    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:47.506911 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.506877    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:47.509108 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.509064    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 29 02:05:47.509251 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.509113    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused"
May 29 02:05:47.509251 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.509157    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused"
May 29 02:05:47.509251 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:47.509111    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 29 02:05:50.506695 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:50.506628    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused"
May 29 02:05:50.507163 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:50.506628    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused"
May 29 02:05:50.507163 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:50.506646    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 29 02:05:50.507404 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:50.507373    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 29 02:05:52.506265 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:52.506229    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:52.506675 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:52.506296    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:52.506675 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:52.506316    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:52.506675 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:52.506328    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:53.507938 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:53.507906    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:53.508313 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:53.508178    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:53.508313 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:53.508274    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:53.512366 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:53.512350    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:54.351986 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:54.351958    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:54.352169 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:54.352013    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:54.352284 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:54.352265    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:05:54.354451 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:05:54.354432    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:06:35.190333 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:35.190294    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b665c8654-4vm8b"]
May 29 02:06:56.869824 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:56.869792    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"]
May 29 02:06:56.870306 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:56.870248    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="registry" containerID="cri-o://1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a" gracePeriod=30
May 29 02:06:56.870397 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:56.870363    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="offline" containerID="cri-o://d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a" gracePeriod=30
May 29 02:06:56.870459 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:56.870387    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="online" containerID="cri-o://600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561" gracePeriod=30
May 29 02:06:56.870516 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:56.870482    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="ui" containerID="cri-o://493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba" gracePeriod=30
May 29 02:06:57.118553 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.118524    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-k625g"]
May 29 02:06:57.122525 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.122461    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.124507 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.124484    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-dmszk\""
May 29 02:06:57.133776 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.133753    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-k625g"]
May 29 02:06:57.218006 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.217964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-online-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.218182 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.218015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-ui-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.218182 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.218038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpnl4\" (UniqueName: \"kubernetes.io/projected/04bfd3da-93ff-4c1c-8d5c-68a470502221-kube-api-access-lpnl4\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.218182 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.218117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-registry-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.218284 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.218192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-offline-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.218284 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.218260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04bfd3da-93ff-4c1c-8d5c-68a470502221-feast-data\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.319439 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.319346    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-online-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.319610 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.319467    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-ui-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.319610 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.319502    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lpnl4\" (UniqueName: \"kubernetes.io/projected/04bfd3da-93ff-4c1c-8d5c-68a470502221-kube-api-access-lpnl4\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.319610 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.319532    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-registry-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.319610 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.319579    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-offline-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.319814 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:06:57.319672    2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 29 02:06:57.319868 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:06:57.319837    2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 29 02:06:57.319920 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:06:57.319905    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-offline-tls podName:04bfd3da-93ff-4c1c-8d5c-68a470502221 nodeName:}" failed. No retries permitted until 2026-05-29 02:06:57.819882439 +0000 UTC m=+514.402873325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-offline-tls") pod "feast-simple-feast-setup-675665f868-k625g" (UID: "04bfd3da-93ff-4c1c-8d5c-68a470502221") : secret "feast-simple-feast-setup-offline-tls" not found
May 29 02:06:57.320077 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.319681    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04bfd3da-93ff-4c1c-8d5c-68a470502221-feast-data\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.320269 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.320010    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04bfd3da-93ff-4c1c-8d5c-68a470502221-feast-data\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.321202 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:06:57.321180    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-ui-tls podName:04bfd3da-93ff-4c1c-8d5c-68a470502221 nodeName:}" failed. No retries permitted until 2026-05-29 02:06:57.821152156 +0000 UTC m=+514.404143055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-ui-tls") pod "feast-simple-feast-setup-675665f868-k625g" (UID: "04bfd3da-93ff-4c1c-8d5c-68a470502221") : secret "feast-simple-feast-setup-ui-tls" not found
May 29 02:06:57.321370 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:06:57.321351    2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 29 02:06:57.321434 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:06:57.321419    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-registry-tls podName:04bfd3da-93ff-4c1c-8d5c-68a470502221 nodeName:}" failed. No retries permitted until 2026-05-29 02:06:57.821401095 +0000 UTC m=+514.404391992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-registry-tls") pod "feast-simple-feast-setup-675665f868-k625g" (UID: "04bfd3da-93ff-4c1c-8d5c-68a470502221") : secret "feast-simple-feast-setup-registry-tls" not found
May 29 02:06:57.323029 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.323004    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-online-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.331358 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.331331    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpnl4\" (UniqueName: \"kubernetes.io/projected/04bfd3da-93ff-4c1c-8d5c-68a470502221-kube-api-access-lpnl4\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.543202 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.543163    2580 generic.go:358] "Generic (PLEG): container finished" podID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerID="493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba" exitCode=0
May 29 02:06:57.543331 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.543237    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerDied","Data":"493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba"}
May 29 02:06:57.826042 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.826013    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-ui-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.826042 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.826045    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-registry-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.826228 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.826072    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-offline-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.828532 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.828498    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-ui-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.828665 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.828545    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-registry-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:57.828708 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:57.828679    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-offline-tls\") pod \"feast-simple-feast-setup-675665f868-k625g\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") " pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:58.031749 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:58.031714    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:06:58.160621 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:58.160579    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-k625g"]
May 29 02:06:58.162606 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:06:58.162568    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04bfd3da_93ff_4c1c_8d5c_68a470502221.slice/crio-0d1f7f3d84d54a7f28ef20df44b2142a7720633fe5942094a955863645f88a0c WatchSource:0}: Error finding container 0d1f7f3d84d54a7f28ef20df44b2142a7720633fe5942094a955863645f88a0c: Status 404 returned error can't find the container with id 0d1f7f3d84d54a7f28ef20df44b2142a7720633fe5942094a955863645f88a0c
May 29 02:06:58.550011 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:58.549975    2580 generic.go:358] "Generic (PLEG): container finished" podID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerID="600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561" exitCode=0
May 29 02:06:58.550186 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:58.550047    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerDied","Data":"600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561"}
May 29 02:06:58.551349 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:58.551325    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerStarted","Data":"a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1"}
May 29 02:06:58.551471 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:06:58.551358    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerStarted","Data":"0d1f7f3d84d54a7f28ef20df44b2142a7720633fe5942094a955863645f88a0c"}
May 29 02:07:00.210326 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.210283    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5b665c8654-4vm8b" podUID="0840238b-c27f-4e12-9d64-85ea466572af" containerName="console" containerID="cri-o://47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f" gracePeriod=15
May 29 02:07:00.447244 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.447222    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b665c8654-4vm8b_0840238b-c27f-4e12-9d64-85ea466572af/console/0.log"
May 29 02:07:00.447350 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.447282    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:07:00.548221 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548191    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-oauth-config\") pod \"0840238b-c27f-4e12-9d64-85ea466572af\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") "
May 29 02:07:00.548221 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548223    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt9d9\" (UniqueName: \"kubernetes.io/projected/0840238b-c27f-4e12-9d64-85ea466572af-kube-api-access-lt9d9\") pod \"0840238b-c27f-4e12-9d64-85ea466572af\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") "
May 29 02:07:00.548432 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548270    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-service-ca\") pod \"0840238b-c27f-4e12-9d64-85ea466572af\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") "
May 29 02:07:00.548489 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548424    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-oauth-serving-cert\") pod \"0840238b-c27f-4e12-9d64-85ea466572af\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") "
May 29 02:07:00.548489 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548472    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-serving-cert\") pod \"0840238b-c27f-4e12-9d64-85ea466572af\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") "
May 29 02:07:00.548620 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548527    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-trusted-ca-bundle\") pod \"0840238b-c27f-4e12-9d64-85ea466572af\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") "
May 29 02:07:00.548620 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548569    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-console-config\") pod \"0840238b-c27f-4e12-9d64-85ea466572af\" (UID: \"0840238b-c27f-4e12-9d64-85ea466572af\") "
May 29 02:07:00.548725 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548678    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-service-ca" (OuterVolumeSpecName: "service-ca") pod "0840238b-c27f-4e12-9d64-85ea466572af" (UID: "0840238b-c27f-4e12-9d64-85ea466572af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:07:00.548881 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548847    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-service-ca\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:00.548969 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548797    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0840238b-c27f-4e12-9d64-85ea466572af" (UID: "0840238b-c27f-4e12-9d64-85ea466572af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:07:00.548969 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548933    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-console-config" (OuterVolumeSpecName: "console-config") pod "0840238b-c27f-4e12-9d64-85ea466572af" (UID: "0840238b-c27f-4e12-9d64-85ea466572af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:07:00.549056 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.548976    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0840238b-c27f-4e12-9d64-85ea466572af" (UID: "0840238b-c27f-4e12-9d64-85ea466572af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:07:00.550657 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.550625    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0840238b-c27f-4e12-9d64-85ea466572af" (UID: "0840238b-c27f-4e12-9d64-85ea466572af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:07:00.550785 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.550764    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0840238b-c27f-4e12-9d64-85ea466572af-kube-api-access-lt9d9" (OuterVolumeSpecName: "kube-api-access-lt9d9") pod "0840238b-c27f-4e12-9d64-85ea466572af" (UID: "0840238b-c27f-4e12-9d64-85ea466572af"). InnerVolumeSpecName "kube-api-access-lt9d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:07:00.550983 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.550960    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0840238b-c27f-4e12-9d64-85ea466572af" (UID: "0840238b-c27f-4e12-9d64-85ea466572af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:07:00.560276 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.560259    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b665c8654-4vm8b_0840238b-c27f-4e12-9d64-85ea466572af/console/0.log"
May 29 02:07:00.560364 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.560293    2580 generic.go:358] "Generic (PLEG): container finished" podID="0840238b-c27f-4e12-9d64-85ea466572af" containerID="47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f" exitCode=2
May 29 02:07:00.560364 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.560324    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b665c8654-4vm8b" event={"ID":"0840238b-c27f-4e12-9d64-85ea466572af","Type":"ContainerDied","Data":"47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f"}
May 29 02:07:00.560484 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.560368    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b665c8654-4vm8b" event={"ID":"0840238b-c27f-4e12-9d64-85ea466572af","Type":"ContainerDied","Data":"ca8176f0e66ea13160f543fbdd17234ce72a53e151091cfa9e8602f4b46742bc"}
May 29 02:07:00.560484 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.560374    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b665c8654-4vm8b"
May 29 02:07:00.560484 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.560391    2580 scope.go:117] "RemoveContainer" containerID="47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f"
May 29 02:07:00.569314 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.569292    2580 scope.go:117] "RemoveContainer" containerID="47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f"
May 29 02:07:00.569647 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:07:00.569616    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f\": container with ID starting with 47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f not found: ID does not exist" containerID="47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f"
May 29 02:07:00.569757 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.569654    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f"} err="failed to get container status \"47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f\": rpc error: code = NotFound desc = could not find container \"47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f\": container with ID starting with 47d3732b4bdfb4f0f1b77e274d7d914796f9ced48a6aa50899694c684290df1f not found: ID does not exist"
May 29 02:07:00.583429 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.583404    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b665c8654-4vm8b"]
May 29 02:07:00.586739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.586718    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b665c8654-4vm8b"]
May 29 02:07:00.649811 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.649774    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-oauth-config\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:00.649811 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.649800    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9d9\" (UniqueName: \"kubernetes.io/projected/0840238b-c27f-4e12-9d64-85ea466572af-kube-api-access-lt9d9\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:00.649811 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.649810    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-oauth-serving-cert\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:00.649811 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.649819    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0840238b-c27f-4e12-9d64-85ea466572af-console-serving-cert\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:00.650135 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.649828    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-trusted-ca-bundle\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:00.650135 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:00.649838    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0840238b-c27f-4e12-9d64-85ea466572af-console-config\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:01.974755 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:01.974720    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0840238b-c27f-4e12-9d64-85ea466572af" path="/var/lib/kubelet/pods/0840238b-c27f-4e12-9d64-85ea466572af/volumes"
May 29 02:07:03.576724 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:03.576695    2580 generic.go:358] "Generic (PLEG): container finished" podID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerID="a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1" exitCode=0
May 29 02:07:03.577059 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:03.576760    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerDied","Data":"a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1"}
May 29 02:07:04.351799 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:04.351737    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 29 02:07:04.351988 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:04.351793    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 29 02:07:04.580699 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:04.580665    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerStarted","Data":"bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506"}
May 29 02:07:09.595434 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:09.595400    2580 generic.go:358] "Generic (PLEG): container finished" podID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerID="bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506" exitCode=0
May 29 02:07:09.595856 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:09.595467    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerDied","Data":"bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506"}
May 29 02:07:10.603454 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:10.603360    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerStarted","Data":"6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718"}
May 29 02:07:10.603454 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:10.603406    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerStarted","Data":"39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c"}
May 29 02:07:10.603454 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:10.603423    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerStarted","Data":"7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b"}
May 29 02:07:10.603454 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:10.603437    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerStarted","Data":"a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d"}
May 29 02:07:10.630434 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:10.630370    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podStartSLOduration=13.63034861 podStartE2EDuration="13.63034861s" podCreationTimestamp="2026-05-29 02:06:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:07:10.62861954 +0000 UTC m=+527.211610446" watchObservedRunningTime="2026-05-29 02:07:10.63034861 +0000 UTC m=+527.213339520"
May 29 02:07:13.032374 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:13.032334    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:13.032951 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:13.032927    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:13.032951 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:13.032954    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:13.033138 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:13.032967    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:13.034671 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:13.034631    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused"
May 29 02:07:13.034785 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:13.034657    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused"
May 29 02:07:13.034785 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:13.034668    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 29 02:07:13.034785 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:13.034744    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 29 02:07:14.351994 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:14.351941    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 29 02:07:14.352576 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:14.351999    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 29 02:07:16.032845 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.032814    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.033290 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.032962    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.033290 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.033248    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.033739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.033697    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.033739 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.033733    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.033939 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.033748    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.033939 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.033905    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.034079 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.034059    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.034195 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.034099    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.037278 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.037261    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.626085 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.626056    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:16.629018 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:16.628997    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:07:24.351955 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:24.351899    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 29 02:07:24.351955 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:24.351942    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 29 02:07:24.352420 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:24.352028    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:07:24.352420 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:24.352080    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:07:27.517011 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.516989    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:07:27.659706 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.659614    2580 generic.go:358] "Generic (PLEG): container finished" podID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerID="d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a" exitCode=137
May 29 02:07:27.659706 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.659646    2580 generic.go:358] "Generic (PLEG): container finished" podID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerID="1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a" exitCode=137
May 29 02:07:27.659926 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.659693    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerDied","Data":"d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a"}
May 29 02:07:27.659926 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.659742    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerDied","Data":"1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a"}
May 29 02:07:27.659926 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.659710    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"
May 29 02:07:27.659926 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.659755    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-sczcv" event={"ID":"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9","Type":"ContainerDied","Data":"b9d54ed28cf5c47213d12cef7308e4d0db43431faa58246365e71ab2a25feb4a"}
May 29 02:07:27.659926 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.659773    2580 scope.go:117] "RemoveContainer" containerID="493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba"
May 29 02:07:27.668075 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.668046    2580 scope.go:117] "RemoveContainer" containerID="d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a"
May 29 02:07:27.675087 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.675071    2580 scope.go:117] "RemoveContainer" containerID="600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561"
May 29 02:07:27.675700 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.675682    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-online-tls\") pod \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") "
May 29 02:07:27.675794 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.675723    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-registry-tls\") pod \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") "
May 29 02:07:27.675794 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.675754    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-feast-data\") pod \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") "
May 29 02:07:27.675895 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.675840    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-offline-tls\") pod \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") "
May 29 02:07:27.675895 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.675865    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glvjm\" (UniqueName: \"kubernetes.io/projected/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-kube-api-access-glvjm\") pod \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") "
May 29 02:07:27.675895 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.675880    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-ui-tls\") pod \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\" (UID: \"dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9\") "
May 29 02:07:27.676491 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.676445    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-feast-data" (OuterVolumeSpecName: "feast-data") pod "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" (UID: "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:07:27.678193 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.678155    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-online-tls" (OuterVolumeSpecName: "online-tls") pod "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" (UID: "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:07:27.678300 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.678271    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-kube-api-access-glvjm" (OuterVolumeSpecName: "kube-api-access-glvjm") pod "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" (UID: "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9"). InnerVolumeSpecName "kube-api-access-glvjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:07:27.678360 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.678348    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" (UID: "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:07:27.678414 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.678367    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" (UID: "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:07:27.678579 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.678562    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" (UID: "dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:07:27.688559 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.688536    2580 scope.go:117] "RemoveContainer" containerID="1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a"
May 29 02:07:27.695506 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.695487    2580 scope.go:117] "RemoveContainer" containerID="836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078"
May 29 02:07:27.702169 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.702152    2580 scope.go:117] "RemoveContainer" containerID="1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410"
May 29 02:07:27.708567 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.708553    2580 scope.go:117] "RemoveContainer" containerID="493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba"
May 29 02:07:27.708792 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:07:27.708777    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba\": container with ID starting with 493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba not found: ID does not exist" containerID="493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba"
May 29 02:07:27.708837 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.708803    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba"} err="failed to get container status \"493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba\": rpc error: code = NotFound desc = could not find container \"493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba\": container with ID starting with 493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba not found: ID does not exist"
May 29 02:07:27.708837 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.708825    2580 scope.go:117] "RemoveContainer" containerID="d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a"
May 29 02:07:27.709023 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:07:27.709009    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a\": container with ID starting with d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a not found: ID does not exist" containerID="d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a"
May 29 02:07:27.709060 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709032    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a"} err="failed to get container status \"d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a\": rpc error: code = NotFound desc = could not find container \"d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a\": container with ID starting with d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a not found: ID does not exist"
May 29 02:07:27.709060 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709043    2580 scope.go:117] "RemoveContainer" containerID="600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561"
May 29 02:07:27.709224 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:07:27.709211    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561\": container with ID starting with 600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561 not found: ID does not exist" containerID="600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561"
May 29 02:07:27.709269 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709227    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561"} err="failed to get container status \"600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561\": rpc error: code = NotFound desc = could not find container \"600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561\": container with ID starting with 600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561 not found: ID does not exist"
May 29 02:07:27.709269 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709239    2580 scope.go:117] "RemoveContainer" containerID="1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a"
May 29 02:07:27.709438 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:07:27.709424    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a\": container with ID starting with 1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a not found: ID does not exist" containerID="1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a"
May 29 02:07:27.709475 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709441    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a"} err="failed to get container status \"1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a\": rpc error: code = NotFound desc = could not find container \"1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a\": container with ID starting with 1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a not found: ID does not exist"
May 29 02:07:27.709475 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709452    2580 scope.go:117] "RemoveContainer" containerID="836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078"
May 29 02:07:27.709634 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:07:27.709616    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078\": container with ID starting with 836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078 not found: ID does not exist" containerID="836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078"
May 29 02:07:27.709727 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709634    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078"} err="failed to get container status \"836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078\": rpc error: code = NotFound desc = could not find container \"836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078\": container with ID starting with 836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078 not found: ID does not exist"
May 29 02:07:27.709727 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709646    2580 scope.go:117] "RemoveContainer" containerID="1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410"
May 29 02:07:27.709881 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:07:27.709862    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410\": container with ID starting with 1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410 not found: ID does not exist" containerID="1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410"
May 29 02:07:27.709941 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709890    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410"} err="failed to get container status \"1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410\": rpc error: code = NotFound desc = could not find container \"1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410\": container with ID starting with 1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410 not found: ID does not exist"
May 29 02:07:27.709941 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.709916    2580 scope.go:117] "RemoveContainer" containerID="493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba"
May 29 02:07:27.710158 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710137    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba"} err="failed to get container status \"493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba\": rpc error: code = NotFound desc = could not find container \"493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba\": container with ID starting with 493d0af2630b1708a20306846f91f3c99752b5eff703002d029140051271f9ba not found: ID does not exist"
May 29 02:07:27.710212 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710159    2580 scope.go:117] "RemoveContainer" containerID="d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a"
May 29 02:07:27.710353 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710336    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a"} err="failed to get container status \"d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a\": rpc error: code = NotFound desc = could not find container \"d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a\": container with ID starting with d78f0ee285e9e2ca67e77a70cd869d7a4318cd02ae290124dbc90933e644c19a not found: ID does not exist"
May 29 02:07:27.710396 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710353    2580 scope.go:117] "RemoveContainer" containerID="600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561"
May 29 02:07:27.710557 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710540    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561"} err="failed to get container status \"600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561\": rpc error: code = NotFound desc = could not find container \"600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561\": container with ID starting with 600cf39139b4c87d6301106ef2b2fe248210a6e83d329ebcb504954645ce8561 not found: ID does not exist"
May 29 02:07:27.710610 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710557    2580 scope.go:117] "RemoveContainer" containerID="1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a"
May 29 02:07:27.710786 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710771    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a"} err="failed to get container status \"1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a\": rpc error: code = NotFound desc = could not find container \"1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a\": container with ID starting with 1cb6911896162f044ded0b70f172f2adde9b0f4f16ac3d0dcf4870180fa91c2a not found: ID does not exist"
May 29 02:07:27.710829 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710786    2580 scope.go:117] "RemoveContainer" containerID="836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078"
May 29 02:07:27.710994 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710975    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078"} err="failed to get container status \"836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078\": rpc error: code = NotFound desc = could not find container \"836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078\": container with ID starting with 836d00563b2451a0100fa5080dc46f37ead608febaa2f01d0f55b6cd959e5078 not found: ID does not exist"
May 29 02:07:27.711038 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.710995    2580 scope.go:117] "RemoveContainer" containerID="1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410"
May 29 02:07:27.711178 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.711161    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410"} err="failed to get container status \"1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410\": rpc error: code = NotFound desc = could not find container \"1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410\": container with ID starting with 1fa5a6352615f25092913672a06fe244ff63a38dbc40eac5044e886f545a7410 not found: ID does not exist"
May 29 02:07:27.776891 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.776866    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:27.776891 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.776889    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:27.776891 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.776897    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:27.777063 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.776906    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glvjm\" (UniqueName: \"kubernetes.io/projected/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-kube-api-access-glvjm\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:27.777063 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.776916    2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-ui-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:27.777063 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.776926    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:07:27.981467 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.981436    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"]
May 29 02:07:27.985863 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:27.985842    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-sczcv"]
May 29 02:07:29.973603 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:07:29.973565    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" path="/var/lib/kubelet/pods/dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9/volumes"
May 29 02:08:18.817517 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817479    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"]
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817793    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="online"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817804    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="online"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817812    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="offline"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817821    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="offline"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817839    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0840238b-c27f-4e12-9d64-85ea466572af" containerName="console"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817845    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0840238b-c27f-4e12-9d64-85ea466572af" containerName="console"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817856    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="feast-apply"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817861    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="feast-apply"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817869    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="feast-init"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817875    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="feast-init"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817881    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="registry"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817886    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="registry"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817893    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="ui"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817900    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="ui"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817948    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="offline"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817958    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="online"
May 29 02:08:18.817959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817964    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="0840238b-c27f-4e12-9d64-85ea466572af" containerName="console"
May 29 02:08:18.818468 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817970    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="ui"
May 29 02:08:18.818468 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.817981    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbf379ab-bfa4-4350-84cf-cbd0c63eb7d9" containerName="registry"
May 29 02:08:18.821203 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.821188    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.823324 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.823299    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 29 02:08:18.823460 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.823336    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 29 02:08:18.823618 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.823579    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 29 02:08:18.823696 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.823631    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 29 02:08:18.823696 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.823644    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 29 02:08:18.823696 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.823655    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 29 02:08:18.823696 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.823649    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-jhqk2\""
May 29 02:08:18.823876 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.823720    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 29 02:08:18.829864 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.829844    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"]
May 29 02:08:18.861878 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.861843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/43cc565b-b253-437c-b304-3d8400946433-feast-data\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.861878 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.861878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-offline-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.862063 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.861963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.862063 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.862018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-ui-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.862063 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.862054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/43cc565b-b253-437c-b304-3d8400946433-registry-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.862160 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.862102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7lr\" (UniqueName: \"kubernetes.io/projected/43cc565b-b253-437c-b304-3d8400946433-kube-api-access-gb7lr\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.962503 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.962461    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-ui-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.962730 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.962510    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/43cc565b-b253-437c-b304-3d8400946433-registry-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.962730 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.962542    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7lr\" (UniqueName: \"kubernetes.io/projected/43cc565b-b253-437c-b304-3d8400946433-kube-api-access-gb7lr\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.962730 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.962586    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/43cc565b-b253-437c-b304-3d8400946433-feast-data\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.962730 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.962637    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-offline-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.962730 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:08:18.962644    2580 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 29 02:08:18.962730 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:08:18.962708    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-ui-tls podName:43cc565b-b253-437c-b304-3d8400946433 nodeName:}" failed. No retries permitted until 2026-05-29 02:08:19.462686864 +0000 UTC m=+596.045677752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-ui-tls") pod "feast-simple-feast-remote-setup-6d859988d6-4sj99" (UID: "43cc565b-b253-437c-b304-3d8400946433") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 29 02:08:18.962730 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.962729    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.963100 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:08:18.962812    2580 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 02:08:18.963100 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:08:18.962852    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls podName:43cc565b-b253-437c-b304-3d8400946433 nodeName:}" failed. No retries permitted until 2026-05-29 02:08:19.462839095 +0000 UTC m=+596.045829978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls") pod "feast-simple-feast-remote-setup-6d859988d6-4sj99" (UID: "43cc565b-b253-437c-b304-3d8400946433") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 02:08:18.963100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.963037    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/43cc565b-b253-437c-b304-3d8400946433-feast-data\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.963292 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.963273    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/43cc565b-b253-437c-b304-3d8400946433-registry-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.965024 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.965004    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-offline-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:18.971076 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:18.971053    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7lr\" (UniqueName: \"kubernetes.io/projected/43cc565b-b253-437c-b304-3d8400946433-kube-api-access-gb7lr\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:19.467752 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:19.467717    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:19.467752 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:19.467757    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-ui-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:19.467957 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:08:19.467861    2580 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 02:08:19.467957 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:08:19.467933    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls podName:43cc565b-b253-437c-b304-3d8400946433 nodeName:}" failed. No retries permitted until 2026-05-29 02:08:20.467917058 +0000 UTC m=+597.050907948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls") pod "feast-simple-feast-remote-setup-6d859988d6-4sj99" (UID: "43cc565b-b253-437c-b304-3d8400946433") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 02:08:19.470078 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:19.470051    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-ui-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:20.477601 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:20.477564    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:20.479981 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:20.479957    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls\") pod \"feast-simple-feast-remote-setup-6d859988d6-4sj99\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:20.631023 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:20.630983    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:20.763529 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:20.763292    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"]
May 29 02:08:20.766017 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:08:20.765987    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43cc565b_b253_437c_b304_3d8400946433.slice/crio-8396b0431c56b44c370f75c9c9fbe9c2c1380b9f5d46dfcee5e7117f55209a65 WatchSource:0}: Error finding container 8396b0431c56b44c370f75c9c9fbe9c2c1380b9f5d46dfcee5e7117f55209a65: Status 404 returned error can't find the container with id 8396b0431c56b44c370f75c9c9fbe9c2c1380b9f5d46dfcee5e7117f55209a65
May 29 02:08:20.809538 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:20.809510    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerStarted","Data":"8396b0431c56b44c370f75c9c9fbe9c2c1380b9f5d46dfcee5e7117f55209a65"}
May 29 02:08:21.813814 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:21.813780    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerStarted","Data":"4341f97623af6f079781b1d8d7906b9e55b64f6622eb8e04a8c7a9478ca971ea"}
May 29 02:08:23.870361 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:23.870328    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:08:23.873509 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:23.873483    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:08:23.876103 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:23.876078    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:08:23.879039 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:23.879019    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:08:25.827147 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:25.827116    2580 generic.go:358] "Generic (PLEG): container finished" podID="43cc565b-b253-437c-b304-3d8400946433" containerID="4341f97623af6f079781b1d8d7906b9e55b64f6622eb8e04a8c7a9478ca971ea" exitCode=0
May 29 02:08:25.827479 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:25.827187    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerDied","Data":"4341f97623af6f079781b1d8d7906b9e55b64f6622eb8e04a8c7a9478ca971ea"}
May 29 02:08:26.831319 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:26.831283    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerStarted","Data":"b978407f7806565f97c63794450e2484c9739e86c2e155252755dda47ecc75a3"}
May 29 02:08:31.846626 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:31.846580    2580 generic.go:358] "Generic (PLEG): container finished" podID="43cc565b-b253-437c-b304-3d8400946433" containerID="b978407f7806565f97c63794450e2484c9739e86c2e155252755dda47ecc75a3" exitCode=0
May 29 02:08:31.847102 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:31.846622    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerDied","Data":"b978407f7806565f97c63794450e2484c9739e86c2e155252755dda47ecc75a3"}
May 29 02:08:31.850438 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:31.850420    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 29 02:08:32.853238 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:32.853203    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerStarted","Data":"c5580b8d5933aa189ea725892e7c251a18783a2f3deada237bda35d1b6aca1aa"}
May 29 02:08:32.853238 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:32.853245    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerStarted","Data":"9aebedd7e202ac444e02e572576efb224ffa8504a1799c4d3153568500a7f83d"}
May 29 02:08:32.853796 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:32.853258    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerStarted","Data":"39d8bcd786b4357468f880cfe333816afe46fc47b1e81294269302d156dc6e46"}
May 29 02:08:32.877779 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:32.877724    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podStartSLOduration=14.877704559 podStartE2EDuration="14.877704559s" podCreationTimestamp="2026-05-29 02:08:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:08:32.876092628 +0000 UTC m=+609.459083534" watchObservedRunningTime="2026-05-29 02:08:32.877704559 +0000 UTC m=+609.460695465"
May 29 02:08:35.631208 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:35.631175    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:35.631757 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:35.631732    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:35.631757 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:35.631760    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:35.633122 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:35.633084    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 29 02:08:35.633223 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:35.633118    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused"
May 29 02:08:35.633223 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:35.633142    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 29 02:08:38.631875 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:38.631844    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:38.632379 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:38.632358    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:38.632456 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:38.632391    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:38.632669 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:38.632651    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:38.636112 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:38.636095    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:38.874603 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:38.874568    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:38.874794 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:38.874632    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:38.875130 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:38.875116    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:08:38.877283 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:08:38.877268    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:09:40.010155 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:40.010121    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"]
May 29 02:09:40.012968 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:40.010611    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="online" containerID="cri-o://39d8bcd786b4357468f880cfe333816afe46fc47b1e81294269302d156dc6e46" gracePeriod=30
May 29 02:09:40.012968 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:40.010664    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="ui" containerID="cri-o://c5580b8d5933aa189ea725892e7c251a18783a2f3deada237bda35d1b6aca1aa" gracePeriod=30
May 29 02:09:40.012968 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:40.010664    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="offline" containerID="cri-o://9aebedd7e202ac444e02e572576efb224ffa8504a1799c4d3153568500a7f83d" gracePeriod=30
May 29 02:09:40.132550 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:40.132509    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-k625g"]
May 29 02:09:40.133018 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:40.132987    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="registry" containerID="cri-o://a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d" gracePeriod=30
May 29 02:09:40.133226 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:40.133037    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="offline" containerID="cri-o://39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c" gracePeriod=30
May 29 02:09:40.133318 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:40.133041    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="ui" containerID="cri-o://6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718" gracePeriod=30
May 29 02:09:40.133381 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:40.133077    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="online" containerID="cri-o://7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b" gracePeriod=30
May 29 02:09:41.055574 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:41.055543    2580 generic.go:358] "Generic (PLEG): container finished" podID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerID="6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718" exitCode=0
May 29 02:09:41.055574 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:41.055572    2580 generic.go:358] "Generic (PLEG): container finished" podID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerID="7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b" exitCode=0
May 29 02:09:41.056049 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:41.055620    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerDied","Data":"6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718"}
May 29 02:09:41.056049 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:41.055656    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerDied","Data":"7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b"}
May 29 02:09:41.057818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:41.057797    2580 generic.go:358] "Generic (PLEG): container finished" podID="43cc565b-b253-437c-b304-3d8400946433" containerID="c5580b8d5933aa189ea725892e7c251a18783a2f3deada237bda35d1b6aca1aa" exitCode=0
May 29 02:09:41.057818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:41.057814    2580 generic.go:358] "Generic (PLEG): container finished" podID="43cc565b-b253-437c-b304-3d8400946433" containerID="39d8bcd786b4357468f880cfe333816afe46fc47b1e81294269302d156dc6e46" exitCode=0
May 29 02:09:41.057970 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:41.057865    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerDied","Data":"c5580b8d5933aa189ea725892e7c251a18783a2f3deada237bda35d1b6aca1aa"}
May 29 02:09:41.057970 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:41.057888    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerDied","Data":"39d8bcd786b4357468f880cfe333816afe46fc47b1e81294269302d156dc6e46"}
May 29 02:09:46.034740 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:46.034698    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 29 02:09:46.626522 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:46.626482    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 29 02:09:48.874865 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:48.874822    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 29 02:09:48.875224 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:48.874834    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 29 02:09:56.034735 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:56.034682    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 29 02:09:56.626557 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:56.626510    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 29 02:09:58.875546 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:58.875501    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 29 02:09:58.875546 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:09:58.875542    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 29 02:10:06.033997 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:06.033951    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 29 02:10:06.034425 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:06.034087    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:10:06.627014 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:06.626970    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 29 02:10:06.627170 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:06.627092    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:10:08.874981 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:08.874942    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 29 02:10:08.875338 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:08.874949    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 29 02:10:08.875338 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:08.875067    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:10:08.875338 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:08.875117    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:10:10.139967 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.139934    2580 generic.go:358] "Generic (PLEG): container finished" podID="43cc565b-b253-437c-b304-3d8400946433" containerID="9aebedd7e202ac444e02e572576efb224ffa8504a1799c4d3153568500a7f83d" exitCode=137
May 29 02:10:10.140368 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.140010    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerDied","Data":"9aebedd7e202ac444e02e572576efb224ffa8504a1799c4d3153568500a7f83d"}
May 29 02:10:10.267041 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.266978    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:10:10.343385 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.343355    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/43cc565b-b253-437c-b304-3d8400946433-feast-data\") pod \"43cc565b-b253-437c-b304-3d8400946433\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") "
May 29 02:10:10.343532 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.343465    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-ui-tls\") pod \"43cc565b-b253-437c-b304-3d8400946433\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") "
May 29 02:10:10.343532 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.343495    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/43cc565b-b253-437c-b304-3d8400946433-registry-tls\") pod \"43cc565b-b253-437c-b304-3d8400946433\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") "
May 29 02:10:10.343532 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.343524    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb7lr\" (UniqueName: \"kubernetes.io/projected/43cc565b-b253-437c-b304-3d8400946433-kube-api-access-gb7lr\") pod \"43cc565b-b253-437c-b304-3d8400946433\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") "
May 29 02:10:10.343714 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.343653    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-offline-tls\") pod \"43cc565b-b253-437c-b304-3d8400946433\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") "
May 29 02:10:10.343773 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.343725    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls\") pod \"43cc565b-b253-437c-b304-3d8400946433\" (UID: \"43cc565b-b253-437c-b304-3d8400946433\") "
May 29 02:10:10.343965 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.343913    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cc565b-b253-437c-b304-3d8400946433-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "43cc565b-b253-437c-b304-3d8400946433" (UID: "43cc565b-b253-437c-b304-3d8400946433"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:10:10.344053 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.343963    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cc565b-b253-437c-b304-3d8400946433-feast-data" (OuterVolumeSpecName: "feast-data") pod "43cc565b-b253-437c-b304-3d8400946433" (UID: "43cc565b-b253-437c-b304-3d8400946433"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:10:10.345829 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.345798    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cc565b-b253-437c-b304-3d8400946433-kube-api-access-gb7lr" (OuterVolumeSpecName: "kube-api-access-gb7lr") pod "43cc565b-b253-437c-b304-3d8400946433" (UID: "43cc565b-b253-437c-b304-3d8400946433"). InnerVolumeSpecName "kube-api-access-gb7lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:10:10.345932 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.345856    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls" (OuterVolumeSpecName: "online-tls") pod "43cc565b-b253-437c-b304-3d8400946433" (UID: "43cc565b-b253-437c-b304-3d8400946433"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:10:10.346018 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.345999    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "43cc565b-b253-437c-b304-3d8400946433" (UID: "43cc565b-b253-437c-b304-3d8400946433"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:10:10.346098 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.346081    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "43cc565b-b253-437c-b304-3d8400946433" (UID: "43cc565b-b253-437c-b304-3d8400946433"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:10:10.448479 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.444570    2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-ui-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.448479 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.444617    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/43cc565b-b253-437c-b304-3d8400946433-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.448479 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.444644    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7lr\" (UniqueName: \"kubernetes.io/projected/43cc565b-b253-437c-b304-3d8400946433-kube-api-access-gb7lr\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.448479 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.444658    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.448479 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.444673    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/43cc565b-b253-437c-b304-3d8400946433-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.448479 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.444687    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/43cc565b-b253-437c-b304-3d8400946433-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.774317 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.774295    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:10:10.846814 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.846783    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpnl4\" (UniqueName: \"kubernetes.io/projected/04bfd3da-93ff-4c1c-8d5c-68a470502221-kube-api-access-lpnl4\") pod \"04bfd3da-93ff-4c1c-8d5c-68a470502221\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") "
May 29 02:10:10.847001 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.846819    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04bfd3da-93ff-4c1c-8d5c-68a470502221-feast-data\") pod \"04bfd3da-93ff-4c1c-8d5c-68a470502221\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") "
May 29 02:10:10.847001 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.846842    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-online-tls\") pod \"04bfd3da-93ff-4c1c-8d5c-68a470502221\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") "
May 29 02:10:10.847001 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.846859    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-offline-tls\") pod \"04bfd3da-93ff-4c1c-8d5c-68a470502221\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") "
May 29 02:10:10.847001 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.846876    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-registry-tls\") pod \"04bfd3da-93ff-4c1c-8d5c-68a470502221\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") "
May 29 02:10:10.847001 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.846900    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-ui-tls\") pod \"04bfd3da-93ff-4c1c-8d5c-68a470502221\" (UID: \"04bfd3da-93ff-4c1c-8d5c-68a470502221\") "
May 29 02:10:10.847469 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.847437    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bfd3da-93ff-4c1c-8d5c-68a470502221-feast-data" (OuterVolumeSpecName: "feast-data") pod "04bfd3da-93ff-4c1c-8d5c-68a470502221" (UID: "04bfd3da-93ff-4c1c-8d5c-68a470502221"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:10:10.849118 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.849090    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "04bfd3da-93ff-4c1c-8d5c-68a470502221" (UID: "04bfd3da-93ff-4c1c-8d5c-68a470502221"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:10:10.849290 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.849272    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "04bfd3da-93ff-4c1c-8d5c-68a470502221" (UID: "04bfd3da-93ff-4c1c-8d5c-68a470502221"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:10:10.849432 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.849406    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-online-tls" (OuterVolumeSpecName: "online-tls") pod "04bfd3da-93ff-4c1c-8d5c-68a470502221" (UID: "04bfd3da-93ff-4c1c-8d5c-68a470502221"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:10:10.849519 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.849432    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "04bfd3da-93ff-4c1c-8d5c-68a470502221" (UID: "04bfd3da-93ff-4c1c-8d5c-68a470502221"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:10:10.849519 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.849456    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bfd3da-93ff-4c1c-8d5c-68a470502221-kube-api-access-lpnl4" (OuterVolumeSpecName: "kube-api-access-lpnl4") pod "04bfd3da-93ff-4c1c-8d5c-68a470502221" (UID: "04bfd3da-93ff-4c1c-8d5c-68a470502221"). InnerVolumeSpecName "kube-api-access-lpnl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:10:10.948022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.947941    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04bfd3da-93ff-4c1c-8d5c-68a470502221-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.948022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.947966    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.948022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.947974    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.948022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.947982    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.948022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.947992    2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/04bfd3da-93ff-4c1c-8d5c-68a470502221-ui-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:10.948022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:10.948000    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpnl4\" (UniqueName: \"kubernetes.io/projected/04bfd3da-93ff-4c1c-8d5c-68a470502221-kube-api-access-lpnl4\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:10:11.147554 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.147522    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"
May 29 02:10:11.148288 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.147516    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99" event={"ID":"43cc565b-b253-437c-b304-3d8400946433","Type":"ContainerDied","Data":"8396b0431c56b44c370f75c9c9fbe9c2c1380b9f5d46dfcee5e7117f55209a65"}
May 29 02:10:11.148288 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.147647    2580 scope.go:117] "RemoveContainer" containerID="c5580b8d5933aa189ea725892e7c251a18783a2f3deada237bda35d1b6aca1aa"
May 29 02:10:11.150260 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.150234    2580 generic.go:358] "Generic (PLEG): container finished" podID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerID="39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c" exitCode=137
May 29 02:10:11.150260 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.150260    2580 generic.go:358] "Generic (PLEG): container finished" podID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerID="a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d" exitCode=137
May 29 02:10:11.150415 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.150268    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerDied","Data":"39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c"}
May 29 02:10:11.150415 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.150304    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerDied","Data":"a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d"}
May 29 02:10:11.150415 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.150321    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g" event={"ID":"04bfd3da-93ff-4c1c-8d5c-68a470502221","Type":"ContainerDied","Data":"0d1f7f3d84d54a7f28ef20df44b2142a7720633fe5942094a955863645f88a0c"}
May 29 02:10:11.150415 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.150320    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-675665f868-k625g"
May 29 02:10:11.157358 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.157339    2580 scope.go:117] "RemoveContainer" containerID="9aebedd7e202ac444e02e572576efb224ffa8504a1799c4d3153568500a7f83d"
May 29 02:10:11.166756 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.166740    2580 scope.go:117] "RemoveContainer" containerID="39d8bcd786b4357468f880cfe333816afe46fc47b1e81294269302d156dc6e46"
May 29 02:10:11.174152 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.174130    2580 scope.go:117] "RemoveContainer" containerID="b978407f7806565f97c63794450e2484c9739e86c2e155252755dda47ecc75a3"
May 29 02:10:11.174339 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.174317    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"]
May 29 02:10:11.178393 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.178374    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6d859988d6-4sj99"]
May 29 02:10:11.181661 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.181645    2580 scope.go:117] "RemoveContainer" containerID="4341f97623af6f079781b1d8d7906b9e55b64f6622eb8e04a8c7a9478ca971ea"
May 29 02:10:11.187978 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.187964    2580 scope.go:117] "RemoveContainer" containerID="6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718"
May 29 02:10:11.190408 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.190386    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-k625g"]
May 29 02:10:11.194174 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.194154    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-675665f868-k625g"]
May 29 02:10:11.196450 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.196433    2580 scope.go:117] "RemoveContainer" containerID="39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c"
May 29 02:10:11.202842 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.202826    2580 scope.go:117] "RemoveContainer" containerID="7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b"
May 29 02:10:11.209123 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.209106    2580 scope.go:117] "RemoveContainer" containerID="a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d"
May 29 02:10:11.215416 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.215399    2580 scope.go:117] "RemoveContainer" containerID="bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506"
May 29 02:10:11.221832 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.221815    2580 scope.go:117] "RemoveContainer" containerID="a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1"
May 29 02:10:11.228307 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.228290    2580 scope.go:117] "RemoveContainer" containerID="6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718"
May 29 02:10:11.228561 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:10:11.228541    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718\": container with ID starting with 6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718 not found: ID does not exist" containerID="6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718"
May 29 02:10:11.228681 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.228575    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718"} err="failed to get container status \"6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718\": rpc error: code = NotFound desc = could not find container \"6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718\": container with ID starting with 6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718 not found: ID does not exist"
May 29 02:10:11.228681 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.228636    2580 scope.go:117] "RemoveContainer" containerID="39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c"
May 29 02:10:11.228886 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:10:11.228868    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c\": container with ID starting with 39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c not found: ID does not exist" containerID="39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c"
May 29 02:10:11.228927 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.228893    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c"} err="failed to get container status \"39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c\": rpc error: code = NotFound desc = could not find container \"39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c\": container with ID starting with 39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c not found: ID does not exist"
May 29 02:10:11.228927 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.228910    2580 scope.go:117] "RemoveContainer" containerID="7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b"
May 29 02:10:11.229126 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:10:11.229112    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b\": container with ID starting with 7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b not found: ID does not exist" containerID="7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b"
May 29 02:10:11.229167 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.229128    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b"} err="failed to get container status \"7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b\": rpc error: code = NotFound desc = could not find container \"7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b\": container with ID starting with 7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b not found: ID does not exist"
May 29 02:10:11.229167 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.229141    2580 scope.go:117] "RemoveContainer" containerID="a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d"
May 29 02:10:11.229314 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:10:11.229297    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d\": container with ID starting with a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d not found: ID does not exist" containerID="a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d"
May 29 02:10:11.229374 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.229323    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d"} err="failed to get container status \"a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d\": rpc error: code = NotFound desc = could not find container \"a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d\": container with ID starting with a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d not found: ID does not exist"
May 29 02:10:11.229374 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.229344    2580 scope.go:117] "RemoveContainer" containerID="bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506"
May 29 02:10:11.229555 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:10:11.229540    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506\": container with ID starting with bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506 not found: ID does not exist" containerID="bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506"
May 29 02:10:11.229608 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.229558    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506"} err="failed to get container status \"bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506\": rpc error: code = NotFound desc = could not find container \"bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506\": container with ID starting with bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506 not found: ID does not exist"
May 29 02:10:11.229608 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.229569    2580 scope.go:117] "RemoveContainer" containerID="a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1"
May 29 02:10:11.229810 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:10:11.229788    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1\": container with ID starting with a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1 not found: ID does not exist" containerID="a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1"
May 29 02:10:11.229891 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.229813    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1"} err="failed to get container status \"a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1\": rpc error: code = NotFound desc = could not find container \"a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1\": container with ID starting with a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1 not found: ID does not exist"
May 29 02:10:11.229891 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.229826    2580 scope.go:117] "RemoveContainer" containerID="6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718"
May 29 02:10:11.230049 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230032    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718"} err="failed to get container status \"6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718\": rpc error: code = NotFound desc = could not find container \"6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718\": container with ID starting with 6a030b959edd9129d2ddfa0c0cb980c77b41cdb26b95634a748f6818e3963718 not found: ID does not exist"
May 29 02:10:11.230095 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230050    2580 scope.go:117] "RemoveContainer" containerID="39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c"
May 29 02:10:11.230249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230233    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c"} err="failed to get container status \"39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c\": rpc error: code = NotFound desc = could not find container \"39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c\": container with ID starting with 39c75b2e7557cf225a891f934bfa7fd1643fbe632e685ee8d8b67800ae4cde5c not found: ID does not exist"
May 29 02:10:11.230290 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230250    2580 scope.go:117] "RemoveContainer" containerID="7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b"
May 29 02:10:11.230484 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230467    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b"} err="failed to get container status \"7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b\": rpc error: code = NotFound desc = could not find container \"7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b\": container with ID starting with 7c31f0d6ca5b1f894191920b77ab0c59be41764d1043179192a5acb79bac815b not found: ID does not exist"
May 29 02:10:11.230484 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230483    2580 scope.go:117] "RemoveContainer" containerID="a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d"
May 29 02:10:11.230686 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230671    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d"} err="failed to get container status \"a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d\": rpc error: code = NotFound desc = could not find container \"a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d\": container with ID starting with a5b2e6e3a1c10279cc753f42c67615eb61bcd1020da0adefb7aa53573158996d not found: ID does not exist"
May 29 02:10:11.230686 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230685    2580 scope.go:117] "RemoveContainer" containerID="bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506"
May 29 02:10:11.230907 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230888    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506"} err="failed to get container status \"bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506\": rpc error: code = NotFound desc = could not find container \"bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506\": container with ID starting with bda1e5ab01fb6161beab5cb973fa286230e54beed461056b25a0d46993587506 not found: ID does not exist"
May 29 02:10:11.230951 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.230909    2580 scope.go:117] "RemoveContainer" containerID="a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1"
May 29 02:10:11.231108 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.231093    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1"} err="failed to get container status \"a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1\": rpc error: code = NotFound desc = could not find container \"a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1\": container with ID starting with a07e75ad25d3cbb6f9cb161fe02248a716010f684438c26ddbf2da6654eabbb1 not found: ID does not exist"
May 29 02:10:11.975037 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.975004    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" path="/var/lib/kubelet/pods/04bfd3da-93ff-4c1c-8d5c-68a470502221/volumes"
May 29 02:10:11.975580 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:11.975568    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43cc565b-b253-437c-b304-3d8400946433" path="/var/lib/kubelet/pods/43cc565b-b253-437c-b304-3d8400946433/volumes"
May 29 02:10:22.998954 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.998920    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-l7vd5"]
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999218    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="registry"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999230    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="registry"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999240    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="feast-apply"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999248    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="feast-apply"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999257    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="online"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999263    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="online"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999270    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="feast-init"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999276    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="feast-init"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999285    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="online"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999290    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="online"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999296    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="offline"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999303    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="offline"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999316    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="feast-init"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999323    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="feast-init"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999333    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="feast-apply"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999338    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="feast-apply"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999344    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="offline"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999348    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="offline"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999354    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="ui"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999358    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="ui"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999364    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="ui"
May 29 02:10:22.999391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999369    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="ui"
May 29 02:10:23.000100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999417    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="online"
May 29 02:10:23.000100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999426    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="offline"
May 29 02:10:23.000100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999433    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="online"
May 29 02:10:23.000100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999439    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="ui"
May 29 02:10:23.000100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999445    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="ui"
May 29 02:10:23.000100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999453    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="04bfd3da-93ff-4c1c-8d5c-68a470502221" containerName="registry"
May 29 02:10:23.000100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:22.999459    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="43cc565b-b253-437c-b304-3d8400946433" containerName="offline"
May 29 02:10:23.002269 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.002250    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:10:23.004407 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.004386    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 02:10:23.004701 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.004671    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 29 02:10:23.004701 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.004671    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 02:10:23.005195 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.005180    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-bstf6\""
May 29 02:10:23.010488 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.010432    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-l7vd5"]
May 29 02:10:23.030526 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.030500    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-854544c579-gc8ht"]
May 29 02:10:23.033382 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.033368    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-gc8ht"
May 29 02:10:23.042991 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.042960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78dl\" (UniqueName: \"kubernetes.io/projected/c99e6383-6160-4adc-afe1-983cb80e127e-kube-api-access-d78dl\") pod \"postgres-7c7656cc6d-l7vd5\" (UID: \"c99e6383-6160-4adc-afe1-983cb80e127e\") " pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:10:23.043135 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.043041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c99e6383-6160-4adc-afe1-983cb80e127e-postgresdata\") pod \"postgres-7c7656cc6d-l7vd5\" (UID: \"c99e6383-6160-4adc-afe1-983cb80e127e\") " pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:10:23.044132 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.044107    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-gc8ht"]
May 29 02:10:23.143655 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.143628    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d78dl\" (UniqueName: \"kubernetes.io/projected/c99e6383-6160-4adc-afe1-983cb80e127e-kube-api-access-d78dl\") pod \"postgres-7c7656cc6d-l7vd5\" (UID: \"c99e6383-6160-4adc-afe1-983cb80e127e\") " pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:10:23.143803 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.143670    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c99e6383-6160-4adc-afe1-983cb80e127e-postgresdata\") pod \"postgres-7c7656cc6d-l7vd5\" (UID: \"c99e6383-6160-4adc-afe1-983cb80e127e\") " pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:10:23.143803 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.143718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh47p\" (UniqueName: \"kubernetes.io/projected/5a6137a5-a379-49a8-b3b1-c390b04a6d43-kube-api-access-hh47p\") pod \"redis-854544c579-gc8ht\" (UID: \"5a6137a5-a379-49a8-b3b1-c390b04a6d43\") " pod="test-ns-feast/redis-854544c579-gc8ht"
May 29 02:10:23.144076 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.144057    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c99e6383-6160-4adc-afe1-983cb80e127e-postgresdata\") pod \"postgres-7c7656cc6d-l7vd5\" (UID: \"c99e6383-6160-4adc-afe1-983cb80e127e\") " pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:10:23.153329 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.153305    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78dl\" (UniqueName: \"kubernetes.io/projected/c99e6383-6160-4adc-afe1-983cb80e127e-kube-api-access-d78dl\") pod \"postgres-7c7656cc6d-l7vd5\" (UID: \"c99e6383-6160-4adc-afe1-983cb80e127e\") " pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:10:23.245116 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.245084    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hh47p\" (UniqueName: \"kubernetes.io/projected/5a6137a5-a379-49a8-b3b1-c390b04a6d43-kube-api-access-hh47p\") pod \"redis-854544c579-gc8ht\" (UID: \"5a6137a5-a379-49a8-b3b1-c390b04a6d43\") " pod="test-ns-feast/redis-854544c579-gc8ht"
May 29 02:10:23.254155 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.254089    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh47p\" (UniqueName: \"kubernetes.io/projected/5a6137a5-a379-49a8-b3b1-c390b04a6d43-kube-api-access-hh47p\") pod \"redis-854544c579-gc8ht\" (UID: \"5a6137a5-a379-49a8-b3b1-c390b04a6d43\") " pod="test-ns-feast/redis-854544c579-gc8ht"
May 29 02:10:23.324995 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.324962    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:10:23.341859 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.341830    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-gc8ht"
May 29 02:10:23.462681 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.462650    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-l7vd5"]
May 29 02:10:23.464382 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:10:23.464356    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc99e6383_6160_4adc_afe1_983cb80e127e.slice/crio-f2555675f0811bf121fd1e427fc8b3da992f77f5db24b193d2ab5350da3091aa WatchSource:0}: Error finding container f2555675f0811bf121fd1e427fc8b3da992f77f5db24b193d2ab5350da3091aa: Status 404 returned error can't find the container with id f2555675f0811bf121fd1e427fc8b3da992f77f5db24b193d2ab5350da3091aa
May 29 02:10:23.465987 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.465967    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 02:10:23.478289 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:23.478255    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-gc8ht"]
May 29 02:10:23.480165 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:10:23.480128    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a6137a5_a379_49a8_b3b1_c390b04a6d43.slice/crio-709ae0258f09bddabf0c1fa883328d74072cf9da2cecb3caedce1da9bee101df WatchSource:0}: Error finding container 709ae0258f09bddabf0c1fa883328d74072cf9da2cecb3caedce1da9bee101df: Status 404 returned error can't find the container with id 709ae0258f09bddabf0c1fa883328d74072cf9da2cecb3caedce1da9bee101df
May 29 02:10:24.197689 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:24.197604    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-gc8ht" event={"ID":"5a6137a5-a379-49a8-b3b1-c390b04a6d43","Type":"ContainerStarted","Data":"709ae0258f09bddabf0c1fa883328d74072cf9da2cecb3caedce1da9bee101df"}
May 29 02:10:24.198903 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:24.198867    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5" event={"ID":"c99e6383-6160-4adc-afe1-983cb80e127e","Type":"ContainerStarted","Data":"f2555675f0811bf121fd1e427fc8b3da992f77f5db24b193d2ab5350da3091aa"}
May 29 02:10:25.217615 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:25.203091    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-gc8ht" event={"ID":"5a6137a5-a379-49a8-b3b1-c390b04a6d43","Type":"ContainerStarted","Data":"a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f"}
May 29 02:10:25.229084 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:25.229027    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/redis-854544c579-gc8ht" podStartSLOduration=0.580019369 podStartE2EDuration="2.229009274s" podCreationTimestamp="2026-05-29 02:10:23 +0000 UTC" firstStartedPulling="2026-05-29 02:10:23.481927857 +0000 UTC m=+720.064918740" lastFinishedPulling="2026-05-29 02:10:25.130917761 +0000 UTC m=+721.713908645" observedRunningTime="2026-05-29 02:10:25.227932191 +0000 UTC m=+721.810923099" watchObservedRunningTime="2026-05-29 02:10:25.229009274 +0000 UTC m=+721.812000180"
May 29 02:10:29.874205 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:29.874183    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 29 02:10:30.220243 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:30.220158    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5" event={"ID":"c99e6383-6160-4adc-afe1-983cb80e127e","Type":"ContainerStarted","Data":"53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e"}
May 29 02:10:30.237058 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:30.237008    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5" podStartSLOduration=1.83086177 podStartE2EDuration="8.236994488s" podCreationTimestamp="2026-05-29 02:10:22 +0000 UTC" firstStartedPulling="2026-05-29 02:10:23.46609245 +0000 UTC m=+720.049083332" lastFinishedPulling="2026-05-29 02:10:29.872225167 +0000 UTC m=+726.455216050" observedRunningTime="2026-05-29 02:10:30.235534885 +0000 UTC m=+726.818525791" watchObservedRunningTime="2026-05-29 02:10:30.236994488 +0000 UTC m=+726.819985394"
May 29 02:10:33.685771 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.685737    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"]
May 29 02:10:33.694693 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.694661    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.697007 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.696980    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 29 02:10:33.697139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.697036    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 29 02:10:33.697139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.696993    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 29 02:10:33.697626 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.697612    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-wvn6n\""
May 29 02:10:33.697684 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.697669    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 29 02:10:33.701411 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.701384    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"]
May 29 02:10:33.726747 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.726722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-online-tls\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.726867 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.726777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5b36f0-4204-4482-b528-b0bca18f0bef-feast-data\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.726867 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.726839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rn7\" (UniqueName: \"kubernetes.io/projected/7c5b36f0-4204-4482-b528-b0bca18f0bef-kube-api-access-79rn7\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.726935 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.726880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-registry-tls\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.726935 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.726895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-offline-tls\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.827721 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.827691    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-registry-tls\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.827721 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.827722    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-offline-tls\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.827889 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.827745    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-online-tls\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.827889 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.827780    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5b36f0-4204-4482-b528-b0bca18f0bef-feast-data\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.827889 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.827804    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-79rn7\" (UniqueName: \"kubernetes.io/projected/7c5b36f0-4204-4482-b528-b0bca18f0bef-kube-api-access-79rn7\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.828275 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.828246    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5b36f0-4204-4482-b528-b0bca18f0bef-feast-data\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.830088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.830062    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-registry-tls\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.830238 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.830217    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-offline-tls\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.830279 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.830217    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-online-tls\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:33.844950 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:33.844927    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rn7\" (UniqueName: \"kubernetes.io/projected/7c5b36f0-4204-4482-b528-b0bca18f0bef-kube-api-access-79rn7\") pod \"feast-credit-scoring-8f96d45b5-n42r5\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") " pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:34.005546 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:34.005466    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:34.144776 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:34.144746    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"]
May 29 02:10:34.147241 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:10:34.147215    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5b36f0_4204_4482_b528_b0bca18f0bef.slice/crio-4f5999b1cca0ebe0225e066cb8754f67c7881701c60f17427488d597ef77a54f WatchSource:0}: Error finding container 4f5999b1cca0ebe0225e066cb8754f67c7881701c60f17427488d597ef77a54f: Status 404 returned error can't find the container with id 4f5999b1cca0ebe0225e066cb8754f67c7881701c60f17427488d597ef77a54f
May 29 02:10:34.233066 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:34.233036    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerStarted","Data":"d70508eb8cc70672a2b7c0fe119e8e1ea7861fed5d166f7a99c0925d568f9adf"}
May 29 02:10:34.233066 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:34.233070    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerStarted","Data":"4f5999b1cca0ebe0225e066cb8754f67c7881701c60f17427488d597ef77a54f"}
May 29 02:10:35.237132 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:35.237099    2580 generic.go:358] "Generic (PLEG): container finished" podID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerID="d70508eb8cc70672a2b7c0fe119e8e1ea7861fed5d166f7a99c0925d568f9adf" exitCode=0
May 29 02:10:35.237490 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:35.237170    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerDied","Data":"d70508eb8cc70672a2b7c0fe119e8e1ea7861fed5d166f7a99c0925d568f9adf"}
May 29 02:10:36.241839 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:36.241805    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerStarted","Data":"27ba84525fb4b875c253cf78a3a7f19c05d4f01e448113af872d96d1546e332d"}
May 29 02:10:42.261424 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:42.261391    2580 generic.go:358] "Generic (PLEG): container finished" podID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerID="27ba84525fb4b875c253cf78a3a7f19c05d4f01e448113af872d96d1546e332d" exitCode=0
May 29 02:10:42.261814 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:42.261442    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerDied","Data":"27ba84525fb4b875c253cf78a3a7f19c05d4f01e448113af872d96d1546e332d"}
May 29 02:10:43.267904 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:43.267871    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerStarted","Data":"046ef113d2a121b9bc00a3bab2430db925b26316f24cc8a50895a4094e2e5d7c"}
May 29 02:10:43.267904 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:43.267908    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerStarted","Data":"ba9258e9b4feae1a166d5e9bcd96bbec9a33cb4487dada91735e8f1abef143d5"}
May 29 02:10:43.268459 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:43.267918    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerStarted","Data":"4b65069a270ec6d0d012f465887e1e162be85fe8adc1844c962600bc7a60c08c"}
May 29 02:10:43.293388 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:43.293336    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podStartSLOduration=10.293317626 podStartE2EDuration="10.293317626s" podCreationTimestamp="2026-05-29 02:10:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:10:43.292735183 +0000 UTC m=+739.875726089" watchObservedRunningTime="2026-05-29 02:10:43.293317626 +0000 UTC m=+739.876308532"
May 29 02:10:44.006294 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:44.006251    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:44.006455 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:44.006305    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:44.006455 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:44.006321    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:46.006243 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:46.006179    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:46.006243 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:46.006224    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:46.006243 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:46.006238    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:46.007989 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:46.007953    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused"
May 29 02:10:46.007989 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:46.007970    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 29 02:10:46.007989 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:46.007987    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused"
May 29 02:10:49.006219 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:49.006167    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused"
May 29 02:10:49.006882 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:49.006860    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:49.007706 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:49.007685    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:49.011959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:49.011937    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:49.296509 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:49.296479    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:52.006530 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:52.006489    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused"
May 29 02:10:55.006520 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:55.006482    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:10:55.007262 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:10:55.007243    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:11:55.082448 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.082418    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"]
May 29 02:11:55.082997 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.082765    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="registry" containerID="cri-o://4b65069a270ec6d0d012f465887e1e162be85fe8adc1844c962600bc7a60c08c" gracePeriod=30
May 29 02:11:55.082997 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.082837    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="offline" containerID="cri-o://046ef113d2a121b9bc00a3bab2430db925b26316f24cc8a50895a4094e2e5d7c" gracePeriod=30
May 29 02:11:55.082997 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.082838    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="online" containerID="cri-o://ba9258e9b4feae1a166d5e9bcd96bbec9a33cb4487dada91735e8f1abef143d5" gracePeriod=30
May 29 02:11:55.098516 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.098489    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-l7vd5"]
May 29 02:11:55.098714 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.098693    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5" podUID="c99e6383-6160-4adc-afe1-983cb80e127e" containerName="postgres" containerID="cri-o://53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e" gracePeriod=30
May 29 02:11:55.105874 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.105847    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-gc8ht"]
May 29 02:11:55.106234 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.106206    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/redis-854544c579-gc8ht" podUID="5a6137a5-a379-49a8-b3b1-c390b04a6d43" containerName="redis" containerID="cri-o://a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f" gracePeriod=30
May 29 02:11:55.374156 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.374131    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-gc8ht"
May 29 02:11:55.377269 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.377248    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:11:55.479968 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.479936    2580 generic.go:358] "Generic (PLEG): container finished" podID="c99e6383-6160-4adc-afe1-983cb80e127e" containerID="53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e" exitCode=0
May 29 02:11:55.480187 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.479986    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5"
May 29 02:11:55.480187 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.480013    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5" event={"ID":"c99e6383-6160-4adc-afe1-983cb80e127e","Type":"ContainerDied","Data":"53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e"}
May 29 02:11:55.480187 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.480057    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-l7vd5" event={"ID":"c99e6383-6160-4adc-afe1-983cb80e127e","Type":"ContainerDied","Data":"f2555675f0811bf121fd1e427fc8b3da992f77f5db24b193d2ab5350da3091aa"}
May 29 02:11:55.480187 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.480079    2580 scope.go:117] "RemoveContainer" containerID="53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e"
May 29 02:11:55.481344 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.481313    2580 generic.go:358] "Generic (PLEG): container finished" podID="5a6137a5-a379-49a8-b3b1-c390b04a6d43" containerID="a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f" exitCode=0
May 29 02:11:55.481430 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.481351    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-gc8ht" event={"ID":"5a6137a5-a379-49a8-b3b1-c390b04a6d43","Type":"ContainerDied","Data":"a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f"}
May 29 02:11:55.481430 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.481362    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-gc8ht"
May 29 02:11:55.481430 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.481373    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-gc8ht" event={"ID":"5a6137a5-a379-49a8-b3b1-c390b04a6d43","Type":"ContainerDied","Data":"709ae0258f09bddabf0c1fa883328d74072cf9da2cecb3caedce1da9bee101df"}
May 29 02:11:55.489558 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.489541    2580 scope.go:117] "RemoveContainer" containerID="53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e"
May 29 02:11:55.489862 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:11:55.489838    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e\": container with ID starting with 53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e not found: ID does not exist" containerID="53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e"
May 29 02:11:55.489918 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.489864    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e"} err="failed to get container status \"53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e\": rpc error: code = NotFound desc = could not find container \"53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e\": container with ID starting with 53c9bb2f68c8ef60ff9a343bfab93502f86ecf2e06307994d287836d8f37d80e not found: ID does not exist"
May 29 02:11:55.489918 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.489882    2580 scope.go:117] "RemoveContainer" containerID="a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f"
May 29 02:11:55.497148 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.497130    2580 scope.go:117] "RemoveContainer" containerID="a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f"
May 29 02:11:55.497427 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:11:55.497399    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f\": container with ID starting with a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f not found: ID does not exist" containerID="a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f"
May 29 02:11:55.497496 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.497431    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f"} err="failed to get container status \"a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f\": rpc error: code = NotFound desc = could not find container \"a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f\": container with ID starting with a71b20f6acb47552871e775e13f48e8a2913aa592b35d4399f7eec5ffbe50e9f not found: ID does not exist"
May 29 02:11:55.506801 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.506775    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh47p\" (UniqueName: \"kubernetes.io/projected/5a6137a5-a379-49a8-b3b1-c390b04a6d43-kube-api-access-hh47p\") pod \"5a6137a5-a379-49a8-b3b1-c390b04a6d43\" (UID: \"5a6137a5-a379-49a8-b3b1-c390b04a6d43\") "
May 29 02:11:55.506889 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.506826    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c99e6383-6160-4adc-afe1-983cb80e127e-postgresdata\") pod \"c99e6383-6160-4adc-afe1-983cb80e127e\" (UID: \"c99e6383-6160-4adc-afe1-983cb80e127e\") "
May 29 02:11:55.506929 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.506906    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d78dl\" (UniqueName: \"kubernetes.io/projected/c99e6383-6160-4adc-afe1-983cb80e127e-kube-api-access-d78dl\") pod \"c99e6383-6160-4adc-afe1-983cb80e127e\" (UID: \"c99e6383-6160-4adc-afe1-983cb80e127e\") "
May 29 02:11:55.508764 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.508734    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6137a5-a379-49a8-b3b1-c390b04a6d43-kube-api-access-hh47p" (OuterVolumeSpecName: "kube-api-access-hh47p") pod "5a6137a5-a379-49a8-b3b1-c390b04a6d43" (UID: "5a6137a5-a379-49a8-b3b1-c390b04a6d43"). InnerVolumeSpecName "kube-api-access-hh47p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:11:55.508866 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.508842    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99e6383-6160-4adc-afe1-983cb80e127e-kube-api-access-d78dl" (OuterVolumeSpecName: "kube-api-access-d78dl") pod "c99e6383-6160-4adc-afe1-983cb80e127e" (UID: "c99e6383-6160-4adc-afe1-983cb80e127e"). InnerVolumeSpecName "kube-api-access-d78dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:11:55.536163 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.536127    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99e6383-6160-4adc-afe1-983cb80e127e-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "c99e6383-6160-4adc-afe1-983cb80e127e" (UID: "c99e6383-6160-4adc-afe1-983cb80e127e"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:11:55.608003 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.607918    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78dl\" (UniqueName: \"kubernetes.io/projected/c99e6383-6160-4adc-afe1-983cb80e127e-kube-api-access-d78dl\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:11:55.608003 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.607949    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh47p\" (UniqueName: \"kubernetes.io/projected/5a6137a5-a379-49a8-b3b1-c390b04a6d43-kube-api-access-hh47p\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:11:55.608003 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.607962    2580 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c99e6383-6160-4adc-afe1-983cb80e127e-postgresdata\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:11:55.822613 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.822566    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-l7vd5"]
May 29 02:11:55.830498 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.830475    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-l7vd5"]
May 29 02:11:55.842804 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.842780    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-gc8ht"]
May 29 02:11:55.847856 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.847837    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-854544c579-gc8ht"]
May 29 02:11:55.974227 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.974145    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a6137a5-a379-49a8-b3b1-c390b04a6d43" path="/var/lib/kubelet/pods/5a6137a5-a379-49a8-b3b1-c390b04a6d43/volumes"
May 29 02:11:55.974484 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:55.974472    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c99e6383-6160-4adc-afe1-983cb80e127e" path="/var/lib/kubelet/pods/c99e6383-6160-4adc-afe1-983cb80e127e/volumes"
May 29 02:11:56.486463 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:56.486423    2580 generic.go:358] "Generic (PLEG): container finished" podID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerID="ba9258e9b4feae1a166d5e9bcd96bbec9a33cb4487dada91735e8f1abef143d5" exitCode=0
May 29 02:11:56.486851 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:56.486493    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerDied","Data":"ba9258e9b4feae1a166d5e9bcd96bbec9a33cb4487dada91735e8f1abef143d5"}
May 29 02:11:59.287444 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:11:59.287406    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 29 02:12:09.287279 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:09.287233    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 29 02:12:19.287277 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:19.287235    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused"
May 29 02:12:19.287754 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:19.287356    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:12:25.570551 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.570515    2580 generic.go:358] "Generic (PLEG): container finished" podID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerID="046ef113d2a121b9bc00a3bab2430db925b26316f24cc8a50895a4094e2e5d7c" exitCode=137
May 29 02:12:25.570551 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.570541    2580 generic.go:358] "Generic (PLEG): container finished" podID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerID="4b65069a270ec6d0d012f465887e1e162be85fe8adc1844c962600bc7a60c08c" exitCode=137
May 29 02:12:25.570973 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.570584    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerDied","Data":"046ef113d2a121b9bc00a3bab2430db925b26316f24cc8a50895a4094e2e5d7c"}
May 29 02:12:25.570973 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.570638    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerDied","Data":"4b65069a270ec6d0d012f465887e1e162be85fe8adc1844c962600bc7a60c08c"}
May 29 02:12:25.720573 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.720552    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:12:25.734864 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.734843    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-online-tls\") pod \"7c5b36f0-4204-4482-b528-b0bca18f0bef\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") "
May 29 02:12:25.734944 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.734877    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79rn7\" (UniqueName: \"kubernetes.io/projected/7c5b36f0-4204-4482-b528-b0bca18f0bef-kube-api-access-79rn7\") pod \"7c5b36f0-4204-4482-b528-b0bca18f0bef\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") "
May 29 02:12:25.734944 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.734909    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5b36f0-4204-4482-b528-b0bca18f0bef-feast-data\") pod \"7c5b36f0-4204-4482-b528-b0bca18f0bef\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") "
May 29 02:12:25.734944 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.734927    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-registry-tls\") pod \"7c5b36f0-4204-4482-b528-b0bca18f0bef\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") "
May 29 02:12:25.735084 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.734977    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-offline-tls\") pod \"7c5b36f0-4204-4482-b528-b0bca18f0bef\" (UID: \"7c5b36f0-4204-4482-b528-b0bca18f0bef\") "
May 29 02:12:25.737005 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.736976    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-online-tls" (OuterVolumeSpecName: "online-tls") pod "7c5b36f0-4204-4482-b528-b0bca18f0bef" (UID: "7c5b36f0-4204-4482-b528-b0bca18f0bef"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:12:25.737116 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.737010    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7c5b36f0-4204-4482-b528-b0bca18f0bef" (UID: "7c5b36f0-4204-4482-b528-b0bca18f0bef"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:12:25.737116 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.737096    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5b36f0-4204-4482-b528-b0bca18f0bef-kube-api-access-79rn7" (OuterVolumeSpecName: "kube-api-access-79rn7") pod "7c5b36f0-4204-4482-b528-b0bca18f0bef" (UID: "7c5b36f0-4204-4482-b528-b0bca18f0bef"). InnerVolumeSpecName "kube-api-access-79rn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:12:25.737305 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.737284    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7c5b36f0-4204-4482-b528-b0bca18f0bef" (UID: "7c5b36f0-4204-4482-b528-b0bca18f0bef"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:12:25.744204 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.744174    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5b36f0-4204-4482-b528-b0bca18f0bef-feast-data" (OuterVolumeSpecName: "feast-data") pod "7c5b36f0-4204-4482-b528-b0bca18f0bef" (UID: "7c5b36f0-4204-4482-b528-b0bca18f0bef"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:12:25.835719 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.835641    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:12:25.835719 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.835672    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:12:25.835719 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.835682    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rn7\" (UniqueName: \"kubernetes.io/projected/7c5b36f0-4204-4482-b528-b0bca18f0bef-kube-api-access-79rn7\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:12:25.835719 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.835694    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5b36f0-4204-4482-b528-b0bca18f0bef-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:12:25.835719 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:25.835708    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c5b36f0-4204-4482-b528-b0bca18f0bef-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:12:26.576103 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:26.576066    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5" event={"ID":"7c5b36f0-4204-4482-b528-b0bca18f0bef","Type":"ContainerDied","Data":"4f5999b1cca0ebe0225e066cb8754f67c7881701c60f17427488d597ef77a54f"}
May 29 02:12:26.576103 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:26.576097    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"
May 29 02:12:26.576652 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:26.576112    2580 scope.go:117] "RemoveContainer" containerID="046ef113d2a121b9bc00a3bab2430db925b26316f24cc8a50895a4094e2e5d7c"
May 29 02:12:26.583800 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:26.583784    2580 scope.go:117] "RemoveContainer" containerID="ba9258e9b4feae1a166d5e9bcd96bbec9a33cb4487dada91735e8f1abef143d5"
May 29 02:12:26.590423 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:26.590406    2580 scope.go:117] "RemoveContainer" containerID="4b65069a270ec6d0d012f465887e1e162be85fe8adc1844c962600bc7a60c08c"
May 29 02:12:26.593175 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:26.593151    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"]
May 29 02:12:26.597418 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:26.597394    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-8f96d45b5-n42r5"]
May 29 02:12:26.598953 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:26.598938    2580 scope.go:117] "RemoveContainer" containerID="27ba84525fb4b875c253cf78a3a7f19c05d4f01e448113af872d96d1546e332d"
May 29 02:12:26.605440 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:26.605423    2580 scope.go:117] "RemoveContainer" containerID="d70508eb8cc70672a2b7c0fe119e8e1ea7861fed5d166f7a99c0925d568f9adf"
May 29 02:12:27.974547 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:12:27.974512    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" path="/var/lib/kubelet/pods/7c5b36f0-4204-4482-b528-b0bca18f0bef/volumes"
May 29 02:13:23.888582 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:13:23.888492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:13:23.891968 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:13:23.891943    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:13:23.901743 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:13:23.901725    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:13:23.905358 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:13:23.905326    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:14:02.236306 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236271    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-d55pl"]
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236775    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="online"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236795    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="online"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236815    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="feast-init"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236824    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="feast-init"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236835    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c99e6383-6160-4adc-afe1-983cb80e127e" containerName="postgres"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236844    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c99e6383-6160-4adc-afe1-983cb80e127e" containerName="postgres"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236852    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="registry"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236861    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="registry"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236870    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="offline"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236878    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="offline"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236894    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="feast-apply"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="feast-apply"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236914    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6137a5-a379-49a8-b3b1-c390b04a6d43" containerName="redis"
May 29 02:14:02.236946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236923    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6137a5-a379-49a8-b3b1-c390b04a6d43" containerName="redis"
May 29 02:14:02.237634 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.236991    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="c99e6383-6160-4adc-afe1-983cb80e127e" containerName="postgres"
May 29 02:14:02.237634 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.237002    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="online"
May 29 02:14:02.237634 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.237012    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="offline"
May 29 02:14:02.237634 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.237024    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="7c5b36f0-4204-4482-b528-b0bca18f0bef" containerName="registry"
May 29 02:14:02.237634 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.237038    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="5a6137a5-a379-49a8-b3b1-c390b04a6d43" containerName="redis"
May 29 02:14:02.240091 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.240070    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-d55pl"
May 29 02:14:02.242537 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.242511    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-rplkt\""
May 29 02:14:02.242537 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.242528    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 29 02:14:02.242959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.242935    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 29 02:14:02.252988 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.252963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rjw\" (UniqueName: \"kubernetes.io/projected/d63a32af-8b78-4f3e-8489-a9d3594c4baf-kube-api-access-c9rjw\") pod \"redis-854544c579-d55pl\" (UID: \"d63a32af-8b78-4f3e-8489-a9d3594c4baf\") " pod="test-ns-feast-rest/redis-854544c579-d55pl"
May 29 02:14:02.254250 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.254228    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-d55pl"]
May 29 02:14:02.354126 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.354090    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rjw\" (UniqueName: \"kubernetes.io/projected/d63a32af-8b78-4f3e-8489-a9d3594c4baf-kube-api-access-c9rjw\") pod \"redis-854544c579-d55pl\" (UID: \"d63a32af-8b78-4f3e-8489-a9d3594c4baf\") " pod="test-ns-feast-rest/redis-854544c579-d55pl"
May 29 02:14:02.362774 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.362750    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rjw\" (UniqueName: \"kubernetes.io/projected/d63a32af-8b78-4f3e-8489-a9d3594c4baf-kube-api-access-c9rjw\") pod \"redis-854544c579-d55pl\" (UID: \"d63a32af-8b78-4f3e-8489-a9d3594c4baf\") " pod="test-ns-feast-rest/redis-854544c579-d55pl"
May 29 02:14:02.562448 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.562418    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-d55pl"
May 29 02:14:02.682324 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.682220    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-d55pl"]
May 29 02:14:02.684864 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:14:02.684837    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd63a32af_8b78_4f3e_8489_a9d3594c4baf.slice/crio-9ff4ef41c52217bad4d244614430ae96ebb781aaab9d49c95efe00edee4a0ab4 WatchSource:0}: Error finding container 9ff4ef41c52217bad4d244614430ae96ebb781aaab9d49c95efe00edee4a0ab4: Status 404 returned error can't find the container with id 9ff4ef41c52217bad4d244614430ae96ebb781aaab9d49c95efe00edee4a0ab4
May 29 02:14:02.853507 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.853419    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-d55pl" event={"ID":"d63a32af-8b78-4f3e-8489-a9d3594c4baf","Type":"ContainerStarted","Data":"e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c"}
May 29 02:14:02.853507 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.853455    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-d55pl" event={"ID":"d63a32af-8b78-4f3e-8489-a9d3594c4baf","Type":"ContainerStarted","Data":"9ff4ef41c52217bad4d244614430ae96ebb781aaab9d49c95efe00edee4a0ab4"}
May 29 02:14:02.866890 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:02.866834    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-d55pl" podStartSLOduration=0.866817975 podStartE2EDuration="866.817975ms" podCreationTimestamp="2026-05-29 02:14:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:14:02.866538403 +0000 UTC m=+939.449529308" watchObservedRunningTime="2026-05-29 02:14:02.866817975 +0000 UTC m=+939.449808880"
May 29 02:14:07.685340 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.685308    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"]
May 29 02:14:07.688302 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.688288    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:14:07.690300 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.690276    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 29 02:14:07.696388 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.696357    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"]
May 29 02:14:07.795042 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.795012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-postgresdata\") pod \"postgres-7c7656cc6d-4kzsx\" (UID: \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:14:07.795220 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.795138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlq97\" (UniqueName: \"kubernetes.io/projected/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-kube-api-access-nlq97\") pod \"postgres-7c7656cc6d-4kzsx\" (UID: \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:14:07.896066 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.896035    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlq97\" (UniqueName: \"kubernetes.io/projected/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-kube-api-access-nlq97\") pod \"postgres-7c7656cc6d-4kzsx\" (UID: \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:14:07.896238 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.896074    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-postgresdata\") pod \"postgres-7c7656cc6d-4kzsx\" (UID: \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:14:07.896469 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.896452    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-postgresdata\") pod \"postgres-7c7656cc6d-4kzsx\" (UID: \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:14:07.904218 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.904189    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlq97\" (UniqueName: \"kubernetes.io/projected/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-kube-api-access-nlq97\") pod \"postgres-7c7656cc6d-4kzsx\" (UID: \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:14:07.998063 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:07.998004    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:14:08.120176 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:08.120150    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"]
May 29 02:14:08.122884 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:14:08.122853    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb85f0b40_ccca_4dee_9e6b_57d64e4cee29.slice/crio-8a6828681e80985a310eb46f7950344b2e3995a57abb5208d5974278b1fcb722 WatchSource:0}: Error finding container 8a6828681e80985a310eb46f7950344b2e3995a57abb5208d5974278b1fcb722: Status 404 returned error can't find the container with id 8a6828681e80985a310eb46f7950344b2e3995a57abb5208d5974278b1fcb722
May 29 02:14:08.870461 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:08.870424    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx" event={"ID":"b85f0b40-ccca-4dee-9e6b-57d64e4cee29","Type":"ContainerStarted","Data":"858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099"}
May 29 02:14:08.870833 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:08.870469    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx" event={"ID":"b85f0b40-ccca-4dee-9e6b-57d64e4cee29","Type":"ContainerStarted","Data":"8a6828681e80985a310eb46f7950344b2e3995a57abb5208d5974278b1fcb722"}
May 29 02:14:08.886747 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:08.886697    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx" podStartSLOduration=1.886678312 podStartE2EDuration="1.886678312s" podCreationTimestamp="2026-05-29 02:14:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:14:08.885673473 +0000 UTC m=+945.468664390" watchObservedRunningTime="2026-05-29 02:14:08.886678312 +0000 UTC m=+945.469669217"
May 29 02:14:13.343181 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.343148    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"]
May 29 02:14:13.346685 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.346666    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.348847 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.348825    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 29 02:14:13.348847 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.348840    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 29 02:14:13.349010 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.348873    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-mw8kg\""
May 29 02:14:13.349010 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.348832    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 29 02:14:13.357020 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.357000    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"]
May 29 02:14:13.434088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.434059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/358e7528-d4ca-4683-9b7a-6d06fb60874e-feast-data\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.434226 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.434106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-registry-tls\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.434226 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.434164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mnx\" (UniqueName: \"kubernetes.io/projected/358e7528-d4ca-4683-9b7a-6d06fb60874e-kube-api-access-p9mnx\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.434226 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.434184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-online-tls\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.534733 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.534699    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-registry-tls\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.534915 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.534750    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mnx\" (UniqueName: \"kubernetes.io/projected/358e7528-d4ca-4683-9b7a-6d06fb60874e-kube-api-access-p9mnx\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.534915 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.534863    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-online-tls\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.535034 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.534924    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/358e7528-d4ca-4683-9b7a-6d06fb60874e-feast-data\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.535311 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.535287    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/358e7528-d4ca-4683-9b7a-6d06fb60874e-feast-data\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.537149 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.537132    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-online-tls\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.537252 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.537236    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-registry-tls\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.541874 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.541857    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mnx\" (UniqueName: \"kubernetes.io/projected/358e7528-d4ca-4683-9b7a-6d06fb60874e-kube-api-access-p9mnx\") pod \"feast-credit-scoring-dd5cb488c-xcjk7\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") " pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.655545 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.655468    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:13.782308 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.782286    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"]
May 29 02:14:13.785169 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:14:13.785123    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358e7528_d4ca_4683_9b7a_6d06fb60874e.slice/crio-99c3bdee2f6cc12ccbd856c6fb48ed3dfc20ea47534a1e329999918f52c58b14 WatchSource:0}: Error finding container 99c3bdee2f6cc12ccbd856c6fb48ed3dfc20ea47534a1e329999918f52c58b14: Status 404 returned error can't find the container with id 99c3bdee2f6cc12ccbd856c6fb48ed3dfc20ea47534a1e329999918f52c58b14
May 29 02:14:13.885765 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.885735    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerStarted","Data":"5b1d2454566eed3b431d9773987e3616c8c5e409bd85302e8c4ef2d109d8ff2e"}
May 29 02:14:13.885885 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:13.885770    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerStarted","Data":"99c3bdee2f6cc12ccbd856c6fb48ed3dfc20ea47534a1e329999918f52c58b14"}
May 29 02:14:15.892307 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:15.892276    2580 generic.go:358] "Generic (PLEG): container finished" podID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerID="5b1d2454566eed3b431d9773987e3616c8c5e409bd85302e8c4ef2d109d8ff2e" exitCode=0
May 29 02:14:15.892714 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:15.892348    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerDied","Data":"5b1d2454566eed3b431d9773987e3616c8c5e409bd85302e8c4ef2d109d8ff2e"}
May 29 02:14:16.897039 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:16.897006    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerStarted","Data":"5ef79b5f9fdefa7136185fbfd573f8969b70d5f694c51a9478953c8bff8b1c98"}
May 29 02:14:23.918748 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:23.918713    2580 generic.go:358] "Generic (PLEG): container finished" podID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerID="5ef79b5f9fdefa7136185fbfd573f8969b70d5f694c51a9478953c8bff8b1c98" exitCode=0
May 29 02:14:23.919101 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:23.918791    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerDied","Data":"5ef79b5f9fdefa7136185fbfd573f8969b70d5f694c51a9478953c8bff8b1c98"}
May 29 02:14:23.922272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:23.922245    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 29 02:14:24.924710 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:24.924659    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerStarted","Data":"2e0a690d2a570b118e1d567676339681065491121065bc6f0e8ff9271029d72d"}
May 29 02:14:24.925069 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:24.924723    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerStarted","Data":"38c8047f50c9d64211a9a8f74fb07743ec16ac3d5253f03b4718b10c736399fd"}
May 29 02:14:24.944958 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:24.944906    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podStartSLOduration=11.944887973 podStartE2EDuration="11.944887973s" podCreationTimestamp="2026-05-29 02:14:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:14:24.943253773 +0000 UTC m=+961.526244678" watchObservedRunningTime="2026-05-29 02:14:24.944887973 +0000 UTC m=+961.527878881"
May 29 02:14:25.655899 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:25.655862    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:25.655899 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:25.655906    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:25.657260 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:25.657225    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused"
May 29 02:14:25.657378 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:25.657265    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 29 02:14:28.656734 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:28.656689    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused"
May 29 02:14:28.657119 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:28.656785    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 29 02:14:31.656165 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:31.656122    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused"
May 29 02:14:31.663680 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:31.663644    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:31.664164 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:31.664138    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:31.667944 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:31.667925    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:33.656302 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:33.656261    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:34.656687 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:34.656639    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused"
May 29 02:14:37.657085 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:37.657049    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:37.657785 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:37.657767    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:14:39.184951 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.184911    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"]
May 29 02:14:39.189646 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.189625    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.192426 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.192402    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 29 02:14:39.192544 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.192428    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-6cmdq\""
May 29 02:14:39.192544 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.192428    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 29 02:14:39.192544 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.192444    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 29 02:14:39.198648 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.198627    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"]
May 29 02:14:39.348368 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.348334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fgc\" (UniqueName: \"kubernetes.io/projected/0159d68c-8146-419a-b3f7-52bd9790b336-kube-api-access-k2fgc\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.348564 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.348373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-registry-tls\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.348564 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.348404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0159d68c-8146-419a-b3f7-52bd9790b336-feast-data\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.348564 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.348525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-online-tls\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.449818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.449732    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fgc\" (UniqueName: \"kubernetes.io/projected/0159d68c-8146-419a-b3f7-52bd9790b336-kube-api-access-k2fgc\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.449818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.449777    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-registry-tls\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.449818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.449807    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0159d68c-8146-419a-b3f7-52bd9790b336-feast-data\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.450029 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.449855    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-online-tls\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.450294 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.450273    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0159d68c-8146-419a-b3f7-52bd9790b336-feast-data\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.452137 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.452110    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-registry-tls\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.452242 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.452193    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-online-tls\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.457671 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.457650    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fgc\" (UniqueName: \"kubernetes.io/projected/0159d68c-8146-419a-b3f7-52bd9790b336-kube-api-access-k2fgc\") pod \"feast-driver-ranking-69b576998d-b6ssr\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") " pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.499829 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.499800    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:39.624261 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.624225    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"]
May 29 02:14:39.627729 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:14:39.627704    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0159d68c_8146_419a_b3f7_52bd9790b336.slice/crio-3c3c684fdc4371ac7c04299250144804b7c8a4ad82183d23084412d74a714ace WatchSource:0}: Error finding container 3c3c684fdc4371ac7c04299250144804b7c8a4ad82183d23084412d74a714ace: Status 404 returned error can't find the container with id 3c3c684fdc4371ac7c04299250144804b7c8a4ad82183d23084412d74a714ace
May 29 02:14:39.977262 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.977229    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerStarted","Data":"afbf58ac8ba08ea5495c7e26294ec48c9da5bd09d8ee42ab7d41ba6e749a6ca8"}
May 29 02:14:39.977262 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:39.977261    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerStarted","Data":"3c3c684fdc4371ac7c04299250144804b7c8a4ad82183d23084412d74a714ace"}
May 29 02:14:44.991530 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:44.991498    2580 generic.go:358] "Generic (PLEG): container finished" podID="0159d68c-8146-419a-b3f7-52bd9790b336" containerID="afbf58ac8ba08ea5495c7e26294ec48c9da5bd09d8ee42ab7d41ba6e749a6ca8" exitCode=0
May 29 02:14:44.991951 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:44.991555    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerDied","Data":"afbf58ac8ba08ea5495c7e26294ec48c9da5bd09d8ee42ab7d41ba6e749a6ca8"}
May 29 02:14:45.995937 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:45.995906    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerStarted","Data":"361ca49a7e31529b0c4daa0e49856dfb0b38a9b9ccb917ab52636b5da6157184"}
May 29 02:14:52.015019 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:52.014985    2580 generic.go:358] "Generic (PLEG): container finished" podID="0159d68c-8146-419a-b3f7-52bd9790b336" containerID="361ca49a7e31529b0c4daa0e49856dfb0b38a9b9ccb917ab52636b5da6157184" exitCode=0
May 29 02:14:52.015377 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:52.015065    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerDied","Data":"361ca49a7e31529b0c4daa0e49856dfb0b38a9b9ccb917ab52636b5da6157184"}
May 29 02:14:53.020873 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:53.020842    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerStarted","Data":"8aa51168bf6c2584d0b307a3257657583f8383673fa5eec28b102b1f07fa557e"}
May 29 02:14:53.020873 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:53.020879    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerStarted","Data":"6152997f83b499b14353272fe4c00a84203f5aec48789f2973b0c5389c19d053"}
May 29 02:14:53.042932 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:53.042868    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podStartSLOduration=14.042851005 podStartE2EDuration="14.042851005s" podCreationTimestamp="2026-05-29 02:14:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:14:53.040822038 +0000 UTC m=+989.623812942" watchObservedRunningTime="2026-05-29 02:14:53.042851005 +0000 UTC m=+989.625842123"
May 29 02:14:54.500750 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:54.500707    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:54.501330 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:54.500767    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:54.502300 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:54.502271    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused"
May 29 02:14:54.502397 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:54.502292    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused"
May 29 02:14:57.501473 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:57.501425    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused"
May 29 02:14:58.120286 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:58.120256    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:58.120799 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:58.120772    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:58.123548 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:58.123527    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:14:59.500031 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:14:59.499994    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:15:00.500964 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:15:00.500925    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused"
May 29 02:15:03.501476 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:15:03.501441    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:15:03.502154 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:15:03.502133    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:16:17.456083 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.456003    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"]
May 29 02:16:17.456607 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.456241    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m" podUID="f6bf796f-3929-41e1-bfed-b56adec38559" containerName="manager" containerID="cri-o://776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854" gracePeriod=10
May 29 02:16:17.725007 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.724980    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"
May 29 02:16:17.752010 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.751978    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"]
May 29 02:16:17.752278 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.752254    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="registry" containerID="cri-o://38c8047f50c9d64211a9a8f74fb07743ec16ac3d5253f03b4718b10c736399fd" gracePeriod=30
May 29 02:16:17.752355 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.752302    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="online" containerID="cri-o://2e0a690d2a570b118e1d567676339681065491121065bc6f0e8ff9271029d72d" gracePeriod=30
May 29 02:16:17.775827 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.775791    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"]
May 29 02:16:17.776206 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.776156    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="registry" containerID="cri-o://6152997f83b499b14353272fe4c00a84203f5aec48789f2973b0c5389c19d053" gracePeriod=30
May 29 02:16:17.776432 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.776271    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="online" containerID="cri-o://8aa51168bf6c2584d0b307a3257657583f8383673fa5eec28b102b1f07fa557e" gracePeriod=30
May 29 02:16:17.828449 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.828420    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c86bs\" (UniqueName: \"kubernetes.io/projected/f6bf796f-3929-41e1-bfed-b56adec38559-kube-api-access-c86bs\") pod \"f6bf796f-3929-41e1-bfed-b56adec38559\" (UID: \"f6bf796f-3929-41e1-bfed-b56adec38559\") "
May 29 02:16:17.830524 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.830491    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bf796f-3929-41e1-bfed-b56adec38559-kube-api-access-c86bs" (OuterVolumeSpecName: "kube-api-access-c86bs") pod "f6bf796f-3929-41e1-bfed-b56adec38559" (UID: "f6bf796f-3929-41e1-bfed-b56adec38559"). InnerVolumeSpecName "kube-api-access-c86bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:16:17.929050 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:17.929014    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c86bs\" (UniqueName: \"kubernetes.io/projected/f6bf796f-3929-41e1-bfed-b56adec38559-kube-api-access-c86bs\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:18.121488 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.121437    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused"
May 29 02:16:18.266966 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.266871    2580 generic.go:358] "Generic (PLEG): container finished" podID="f6bf796f-3929-41e1-bfed-b56adec38559" containerID="776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854" exitCode=0
May 29 02:16:18.266966 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.266932    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m" event={"ID":"f6bf796f-3929-41e1-bfed-b56adec38559","Type":"ContainerDied","Data":"776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854"}
May 29 02:16:18.266966 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.266965    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m" event={"ID":"f6bf796f-3929-41e1-bfed-b56adec38559","Type":"ContainerDied","Data":"f39f2b0b524e7dcd404e3061a3d4749ba7c599f661e0a55e6d35bb5e58c36c43"}
May 29 02:16:18.267199 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.266980    2580 scope.go:117] "RemoveContainer" containerID="776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854"
May 29 02:16:18.267199 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.266988    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"
May 29 02:16:18.276549 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.276528    2580 scope.go:117] "RemoveContainer" containerID="776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854"
May 29 02:16:18.276858 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:16:18.276829    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854\": container with ID starting with 776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854 not found: ID does not exist" containerID="776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854"
May 29 02:16:18.276942 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.276862    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854"} err="failed to get container status \"776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854\": rpc error: code = NotFound desc = could not find container \"776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854\": container with ID starting with 776ddd558aae98b977063e515b935f39a5882cd34f63aba82c0c7e19eb750854 not found: ID does not exist"
May 29 02:16:18.281832 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.281811    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"]
May 29 02:16:18.285261 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:18.285233    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7b4c9ffb8c-fp49m"]
May 29 02:16:19.272563 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:19.272528    2580 generic.go:358] "Generic (PLEG): container finished" podID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerID="2e0a690d2a570b118e1d567676339681065491121065bc6f0e8ff9271029d72d" exitCode=0
May 29 02:16:19.272966 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:19.272619    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerDied","Data":"2e0a690d2a570b118e1d567676339681065491121065bc6f0e8ff9271029d72d"}
May 29 02:16:19.274401 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:19.274377    2580 generic.go:358] "Generic (PLEG): container finished" podID="0159d68c-8146-419a-b3f7-52bd9790b336" containerID="8aa51168bf6c2584d0b307a3257657583f8383673fa5eec28b102b1f07fa557e" exitCode=0
May 29 02:16:19.274497 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:19.274446    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerDied","Data":"8aa51168bf6c2584d0b307a3257657583f8383673fa5eec28b102b1f07fa557e"}
May 29 02:16:19.864850 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:19.864817    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"]
May 29 02:16:19.865053 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:19.865017    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx" podUID="b85f0b40-ccca-4dee-9e6b-57d64e4cee29" containerName="postgres" containerID="cri-o://858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099" gracePeriod=30
May 29 02:16:19.869270 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:19.869246    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-d55pl"]
May 29 02:16:19.869450 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:19.869428    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-d55pl" podUID="d63a32af-8b78-4f3e-8489-a9d3594c4baf" containerName="redis" containerID="cri-o://e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c" gracePeriod=30
May 29 02:16:19.975147 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:19.975115    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6bf796f-3929-41e1-bfed-b56adec38559" path="/var/lib/kubelet/pods/f6bf796f-3929-41e1-bfed-b56adec38559/volumes"
May 29 02:16:20.132744 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.132722    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:16:20.135891 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.135871    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-d55pl"
May 29 02:16:20.247662 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.247628    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9rjw\" (UniqueName: \"kubernetes.io/projected/d63a32af-8b78-4f3e-8489-a9d3594c4baf-kube-api-access-c9rjw\") pod \"d63a32af-8b78-4f3e-8489-a9d3594c4baf\" (UID: \"d63a32af-8b78-4f3e-8489-a9d3594c4baf\") "
May 29 02:16:20.247829 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.247681    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlq97\" (UniqueName: \"kubernetes.io/projected/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-kube-api-access-nlq97\") pod \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\" (UID: \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\") "
May 29 02:16:20.247829 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.247710    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-postgresdata\") pod \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\" (UID: \"b85f0b40-ccca-4dee-9e6b-57d64e4cee29\") "
May 29 02:16:20.249809 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.249769    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-kube-api-access-nlq97" (OuterVolumeSpecName: "kube-api-access-nlq97") pod "b85f0b40-ccca-4dee-9e6b-57d64e4cee29" (UID: "b85f0b40-ccca-4dee-9e6b-57d64e4cee29"). InnerVolumeSpecName "kube-api-access-nlq97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:16:20.249925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.249802    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63a32af-8b78-4f3e-8489-a9d3594c4baf-kube-api-access-c9rjw" (OuterVolumeSpecName: "kube-api-access-c9rjw") pod "d63a32af-8b78-4f3e-8489-a9d3594c4baf" (UID: "d63a32af-8b78-4f3e-8489-a9d3594c4baf"). InnerVolumeSpecName "kube-api-access-c9rjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:16:20.277546 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.277516    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "b85f0b40-ccca-4dee-9e6b-57d64e4cee29" (UID: "b85f0b40-ccca-4dee-9e6b-57d64e4cee29"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:16:20.278708 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.278684    2580 generic.go:358] "Generic (PLEG): container finished" podID="d63a32af-8b78-4f3e-8489-a9d3594c4baf" containerID="e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c" exitCode=0
May 29 02:16:20.278804 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.278761    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-d55pl"
May 29 02:16:20.278804 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.278763    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-d55pl" event={"ID":"d63a32af-8b78-4f3e-8489-a9d3594c4baf","Type":"ContainerDied","Data":"e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c"}
May 29 02:16:20.278804 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.278799    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-d55pl" event={"ID":"d63a32af-8b78-4f3e-8489-a9d3594c4baf","Type":"ContainerDied","Data":"9ff4ef41c52217bad4d244614430ae96ebb781aaab9d49c95efe00edee4a0ab4"}
May 29 02:16:20.278933 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.278819    2580 scope.go:117] "RemoveContainer" containerID="e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c"
May 29 02:16:20.279987 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.279961    2580 generic.go:358] "Generic (PLEG): container finished" podID="b85f0b40-ccca-4dee-9e6b-57d64e4cee29" containerID="858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099" exitCode=0
May 29 02:16:20.280099 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.280003    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx" event={"ID":"b85f0b40-ccca-4dee-9e6b-57d64e4cee29","Type":"ContainerDied","Data":"858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099"}
May 29 02:16:20.280099 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.280016    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"
May 29 02:16:20.280099 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.280022    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-4kzsx" event={"ID":"b85f0b40-ccca-4dee-9e6b-57d64e4cee29","Type":"ContainerDied","Data":"8a6828681e80985a310eb46f7950344b2e3995a57abb5208d5974278b1fcb722"}
May 29 02:16:20.287316 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.287256    2580 scope.go:117] "RemoveContainer" containerID="e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c"
May 29 02:16:20.287579 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:16:20.287554    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c\": container with ID starting with e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c not found: ID does not exist" containerID="e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c"
May 29 02:16:20.287827 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.287586    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c"} err="failed to get container status \"e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c\": rpc error: code = NotFound desc = could not find container \"e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c\": container with ID starting with e6887826bbfcad275492f4519af5d4a598e34ac58f72958638fd73bf75c1ba8c not found: ID does not exist"
May 29 02:16:20.287827 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.287631    2580 scope.go:117] "RemoveContainer" containerID="858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099"
May 29 02:16:20.295998 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.295984    2580 scope.go:117] "RemoveContainer" containerID="858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099"
May 29 02:16:20.296224 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:16:20.296207    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099\": container with ID starting with 858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099 not found: ID does not exist" containerID="858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099"
May 29 02:16:20.296269 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.296231    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099"} err="failed to get container status \"858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099\": rpc error: code = NotFound desc = could not find container \"858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099\": container with ID starting with 858a4c43d37cf4a313ffb1b9bbafee29ab01b1fb918abcdcb5fda488744d3099 not found: ID does not exist"
May 29 02:16:20.300920 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.300898    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-d55pl"]
May 29 02:16:20.306007 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.305987    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-d55pl"]
May 29 02:16:20.315897 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.315878    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"]
May 29 02:16:20.319708 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.319689    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-4kzsx"]
May 29 02:16:20.349013 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.348991    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlq97\" (UniqueName: \"kubernetes.io/projected/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-kube-api-access-nlq97\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:20.349013 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.349012    2580 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b85f0b40-ccca-4dee-9e6b-57d64e4cee29-postgresdata\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:20.349123 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:20.349023    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9rjw\" (UniqueName: \"kubernetes.io/projected/d63a32af-8b78-4f3e-8489-a9d3594c4baf-kube-api-access-c9rjw\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:21.664421 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:21.664378    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 29 02:16:21.974102 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:21.974029    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85f0b40-ccca-4dee-9e6b-57d64e4cee29" path="/var/lib/kubelet/pods/b85f0b40-ccca-4dee-9e6b-57d64e4cee29/volumes"
May 29 02:16:21.974328 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:21.974316    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d63a32af-8b78-4f3e-8489-a9d3594c4baf" path="/var/lib/kubelet/pods/d63a32af-8b78-4f3e-8489-a9d3594c4baf/volumes"
May 29 02:16:28.121105 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:28.121057    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused"
May 29 02:16:31.665181 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:31.665136    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 29 02:16:38.120947 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:38.120906    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused"
May 29 02:16:38.121331 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:38.121018    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:16:41.664694 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:41.664655    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused"
May 29 02:16:41.665075 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:41.664777    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:16:48.121876 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.121838    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused"
May 29 02:16:48.365866 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.365835    2580 generic.go:358] "Generic (PLEG): container finished" podID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerID="38c8047f50c9d64211a9a8f74fb07743ec16ac3d5253f03b4718b10c736399fd" exitCode=137
May 29 02:16:48.366099 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.365883    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerDied","Data":"38c8047f50c9d64211a9a8f74fb07743ec16ac3d5253f03b4718b10c736399fd"}
May 29 02:16:48.368357 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.368329    2580 generic.go:358] "Generic (PLEG): container finished" podID="0159d68c-8146-419a-b3f7-52bd9790b336" containerID="6152997f83b499b14353272fe4c00a84203f5aec48789f2973b0c5389c19d053" exitCode=137
May 29 02:16:48.368455 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.368371    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerDied","Data":"6152997f83b499b14353272fe4c00a84203f5aec48789f2973b0c5389c19d053"}
May 29 02:16:48.399818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.399795    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:16:48.434112 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.434092    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:16:48.438303 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.438280    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2fgc\" (UniqueName: \"kubernetes.io/projected/0159d68c-8146-419a-b3f7-52bd9790b336-kube-api-access-k2fgc\") pod \"0159d68c-8146-419a-b3f7-52bd9790b336\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") "
May 29 02:16:48.438420 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.438311    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/358e7528-d4ca-4683-9b7a-6d06fb60874e-feast-data\") pod \"358e7528-d4ca-4683-9b7a-6d06fb60874e\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") "
May 29 02:16:48.438420 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.438341    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-registry-tls\") pod \"358e7528-d4ca-4683-9b7a-6d06fb60874e\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") "
May 29 02:16:48.438420 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.438368    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-registry-tls\") pod \"0159d68c-8146-419a-b3f7-52bd9790b336\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") "
May 29 02:16:48.438420 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.438397    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-online-tls\") pod \"0159d68c-8146-419a-b3f7-52bd9790b336\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") "
May 29 02:16:48.438661 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.438429    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0159d68c-8146-419a-b3f7-52bd9790b336-feast-data\") pod \"0159d68c-8146-419a-b3f7-52bd9790b336\" (UID: \"0159d68c-8146-419a-b3f7-52bd9790b336\") "
May 29 02:16:48.438661 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.438476    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9mnx\" (UniqueName: \"kubernetes.io/projected/358e7528-d4ca-4683-9b7a-6d06fb60874e-kube-api-access-p9mnx\") pod \"358e7528-d4ca-4683-9b7a-6d06fb60874e\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") "
May 29 02:16:48.438661 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.438518    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-online-tls\") pod \"358e7528-d4ca-4683-9b7a-6d06fb60874e\" (UID: \"358e7528-d4ca-4683-9b7a-6d06fb60874e\") "
May 29 02:16:48.439349 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.439305    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0159d68c-8146-419a-b3f7-52bd9790b336-feast-data" (OuterVolumeSpecName: "feast-data") pod "0159d68c-8146-419a-b3f7-52bd9790b336" (UID: "0159d68c-8146-419a-b3f7-52bd9790b336"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:16:48.440920 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.440883    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0159d68c-8146-419a-b3f7-52bd9790b336" (UID: "0159d68c-8146-419a-b3f7-52bd9790b336"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:16:48.441336 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.441309    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0159d68c-8146-419a-b3f7-52bd9790b336-kube-api-access-k2fgc" (OuterVolumeSpecName: "kube-api-access-k2fgc") pod "0159d68c-8146-419a-b3f7-52bd9790b336" (UID: "0159d68c-8146-419a-b3f7-52bd9790b336"). InnerVolumeSpecName "kube-api-access-k2fgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:16:48.441444 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.441421    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "358e7528-d4ca-4683-9b7a-6d06fb60874e" (UID: "358e7528-d4ca-4683-9b7a-6d06fb60874e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:16:48.441504 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.441465    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-online-tls" (OuterVolumeSpecName: "online-tls") pod "358e7528-d4ca-4683-9b7a-6d06fb60874e" (UID: "358e7528-d4ca-4683-9b7a-6d06fb60874e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:16:48.441711 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.441676    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358e7528-d4ca-4683-9b7a-6d06fb60874e-kube-api-access-p9mnx" (OuterVolumeSpecName: "kube-api-access-p9mnx") pod "358e7528-d4ca-4683-9b7a-6d06fb60874e" (UID: "358e7528-d4ca-4683-9b7a-6d06fb60874e"). InnerVolumeSpecName "kube-api-access-p9mnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:16:48.441711 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.441683    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-online-tls" (OuterVolumeSpecName: "online-tls") pod "0159d68c-8146-419a-b3f7-52bd9790b336" (UID: "0159d68c-8146-419a-b3f7-52bd9790b336"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:16:48.448120 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.448094    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358e7528-d4ca-4683-9b7a-6d06fb60874e-feast-data" (OuterVolumeSpecName: "feast-data") pod "358e7528-d4ca-4683-9b7a-6d06fb60874e" (UID: "358e7528-d4ca-4683-9b7a-6d06fb60874e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:16:48.539628 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.539582    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0159d68c-8146-419a-b3f7-52bd9790b336-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:48.539628 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.539628    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9mnx\" (UniqueName: \"kubernetes.io/projected/358e7528-d4ca-4683-9b7a-6d06fb60874e-kube-api-access-p9mnx\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:48.539818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.539639    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:48.539818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.539648    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fgc\" (UniqueName: \"kubernetes.io/projected/0159d68c-8146-419a-b3f7-52bd9790b336-kube-api-access-k2fgc\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:48.539818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.539657    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/358e7528-d4ca-4683-9b7a-6d06fb60874e-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:48.539818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.539665    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/358e7528-d4ca-4683-9b7a-6d06fb60874e-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:48.539818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.539672    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:48.539818 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:48.539681    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0159d68c-8146-419a-b3f7-52bd9790b336-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:16:49.373366 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.373326    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7" event={"ID":"358e7528-d4ca-4683-9b7a-6d06fb60874e","Type":"ContainerDied","Data":"99c3bdee2f6cc12ccbd856c6fb48ed3dfc20ea47534a1e329999918f52c58b14"}
May 29 02:16:49.373366 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.373354    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"
May 29 02:16:49.373830 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.373375    2580 scope.go:117] "RemoveContainer" containerID="2e0a690d2a570b118e1d567676339681065491121065bc6f0e8ff9271029d72d"
May 29 02:16:49.375383 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.375353    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr" event={"ID":"0159d68c-8146-419a-b3f7-52bd9790b336","Type":"ContainerDied","Data":"3c3c684fdc4371ac7c04299250144804b7c8a4ad82183d23084412d74a714ace"}
May 29 02:16:49.375471 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.375402    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"
May 29 02:16:49.381352 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.381328    2580 scope.go:117] "RemoveContainer" containerID="38c8047f50c9d64211a9a8f74fb07743ec16ac3d5253f03b4718b10c736399fd"
May 29 02:16:49.389724 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.389703    2580 scope.go:117] "RemoveContainer" containerID="5ef79b5f9fdefa7136185fbfd573f8969b70d5f694c51a9478953c8bff8b1c98"
May 29 02:16:49.396922 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.396900    2580 scope.go:117] "RemoveContainer" containerID="5b1d2454566eed3b431d9773987e3616c8c5e409bd85302e8c4ef2d109d8ff2e"
May 29 02:16:49.397197 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.397142    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"]
May 29 02:16:49.403011 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.402983    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-dd5cb488c-xcjk7"]
May 29 02:16:49.404347 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.404329    2580 scope.go:117] "RemoveContainer" containerID="8aa51168bf6c2584d0b307a3257657583f8383673fa5eec28b102b1f07fa557e"
May 29 02:16:49.410994 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.410979    2580 scope.go:117] "RemoveContainer" containerID="6152997f83b499b14353272fe4c00a84203f5aec48789f2973b0c5389c19d053"
May 29 02:16:49.413215 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.413192    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"]
May 29 02:16:49.418942 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.418914    2580 scope.go:117] "RemoveContainer" containerID="361ca49a7e31529b0c4daa0e49856dfb0b38a9b9ccb917ab52636b5da6157184"
May 29 02:16:49.419462 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.419441    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69b576998d-b6ssr"]
May 29 02:16:49.425465 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.425450    2580 scope.go:117] "RemoveContainer" containerID="afbf58ac8ba08ea5495c7e26294ec48c9da5bd09d8ee42ab7d41ba6e749a6ca8"
May 29 02:16:49.973753 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.973715    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" path="/var/lib/kubelet/pods/0159d68c-8146-419a-b3f7-52bd9790b336/volumes"
May 29 02:16:49.974197 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:49.974185    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" path="/var/lib/kubelet/pods/358e7528-d4ca-4683-9b7a-6d06fb60874e/volumes"
May 29 02:16:54.449378 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449347    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"]
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449628    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85f0b40-ccca-4dee-9e6b-57d64e4cee29" containerName="postgres"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449640    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85f0b40-ccca-4dee-9e6b-57d64e4cee29" containerName="postgres"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449649    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="online"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="online"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449668    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="registry"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449673    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="registry"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449682    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d63a32af-8b78-4f3e-8489-a9d3594c4baf" containerName="redis"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449688    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d63a32af-8b78-4f3e-8489-a9d3594c4baf" containerName="redis"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449697    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="feast-apply"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449705    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="feast-apply"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449714    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="registry"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449719    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="registry"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449725    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6bf796f-3929-41e1-bfed-b56adec38559" containerName="manager"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449730    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6bf796f-3929-41e1-bfed-b56adec38559" containerName="manager"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449737    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="feast-init"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449743    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="feast-init"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449750    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="feast-init"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449755    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="feast-init"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449760    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="feast-apply"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449767    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="feast-apply"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449774    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="online"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449782    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="online"
May 29 02:16:54.449826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449834    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="b85f0b40-ccca-4dee-9e6b-57d64e4cee29" containerName="postgres"
May 29 02:16:54.450538 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449844    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="d63a32af-8b78-4f3e-8489-a9d3594c4baf" containerName="redis"
May 29 02:16:54.450538 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449854    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="registry"
May 29 02:16:54.450538 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449862    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="f6bf796f-3929-41e1-bfed-b56adec38559" containerName="manager"
May 29 02:16:54.450538 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449873    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="online"
May 29 02:16:54.450538 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449879    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="0159d68c-8146-419a-b3f7-52bd9790b336" containerName="online"
May 29 02:16:54.450538 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.449885    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="358e7528-d4ca-4683-9b7a-6d06fb60874e" containerName="registry"
May 29 02:16:54.455295 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.455278    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.457604 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.457567    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 29 02:16:54.457729 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.457672    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 29 02:16:54.457971 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.457951    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 29 02:16:54.458088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.458010    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 02:16:54.458088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.458060    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-g526h\""
May 29 02:16:54.458200 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.458017    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 02:16:54.458200 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.458097    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 29 02:16:54.466427 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.466409    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"]
May 29 02:16:54.484034 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.484013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfzpf\" (UniqueName: \"kubernetes.io/projected/60d21940-92ac-4e36-918e-04ce83738a8a-kube-api-access-dfzpf\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.484125 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.484051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.484125 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.484079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.484125 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.484095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60d21940-92ac-4e36-918e-04ce83738a8a-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.484242 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.484127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.484242 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.484145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.584681 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.584652    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.584846 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.584690    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.584846 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.584707    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60d21940-92ac-4e36-918e-04ce83738a8a-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.584846 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.584731    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.584846 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.584753    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.584846 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.584787    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dfzpf\" (UniqueName: \"kubernetes.io/projected/60d21940-92ac-4e36-918e-04ce83738a8a-kube-api-access-dfzpf\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.585097 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.585061    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60d21940-92ac-4e36-918e-04ce83738a8a-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.587216 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.587193    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.587315 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.587255    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.587315 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.587304    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.587428 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.587320    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.592221 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.592202    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfzpf\" (UniqueName: \"kubernetes.io/projected/60d21940-92ac-4e36-918e-04ce83738a8a-kube-api-access-dfzpf\") pod \"feast-simple-feast-setup-664fcfcc58-qlj7v\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.776803 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.776704    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:16:54.905587 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.905555    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"]
May 29 02:16:54.909017 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:16:54.908993    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60d21940_92ac_4e36_918e_04ce83738a8a.slice/crio-820adad9c84966ca5dabec73024f806355e6bd902272dc79c250c3d43bfc00ed WatchSource:0}: Error finding container 820adad9c84966ca5dabec73024f806355e6bd902272dc79c250c3d43bfc00ed: Status 404 returned error can't find the container with id 820adad9c84966ca5dabec73024f806355e6bd902272dc79c250c3d43bfc00ed
May 29 02:16:54.910725 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:54.910708    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 02:16:55.396831 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:16:55.396799    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerStarted","Data":"820adad9c84966ca5dabec73024f806355e6bd902272dc79c250c3d43bfc00ed"}
May 29 02:17:11.448748 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:11.448712    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerStarted","Data":"5ff7c2921458e78b4d4626a5534aacc4fdba321dafbdda4ee0a663f16e5daf52"}
May 29 02:17:15.460154 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:15.460066    2580 generic.go:358] "Generic (PLEG): container finished" podID="60d21940-92ac-4e36-918e-04ce83738a8a" containerID="5ff7c2921458e78b4d4626a5534aacc4fdba321dafbdda4ee0a663f16e5daf52" exitCode=0
May 29 02:17:15.460485 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:15.460146    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerDied","Data":"5ff7c2921458e78b4d4626a5534aacc4fdba321dafbdda4ee0a663f16e5daf52"}
May 29 02:17:16.464944 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:16.464904    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerStarted","Data":"f5496c5313a7ba849e1f2fd0e7bbbd46811732e5d51c35c04ecf60f839ff2b26"}
May 29 02:17:21.487471 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:21.487437    2580 generic.go:358] "Generic (PLEG): container finished" podID="60d21940-92ac-4e36-918e-04ce83738a8a" containerID="f5496c5313a7ba849e1f2fd0e7bbbd46811732e5d51c35c04ecf60f839ff2b26" exitCode=0
May 29 02:17:21.487900 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:21.487504    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerDied","Data":"f5496c5313a7ba849e1f2fd0e7bbbd46811732e5d51c35c04ecf60f839ff2b26"}
May 29 02:17:22.495172 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:22.495129    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerStarted","Data":"250ec128a9e675a485d2476c936f9f883f63f8c2e802740da86b2f13f92dc386"}
May 29 02:17:22.495673 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:22.495180    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerStarted","Data":"121ac1501890daf2ee31d0b5628d8f79ab8ebf401fe77f0f0b6b20f98f2569d4"}
May 29 02:17:22.495673 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:22.495194    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerStarted","Data":"ad5b107ddd4ce8721ad075e2dc5a03bfd666869d66bd1129e1dcde0c18caa5e6"}
May 29 02:17:22.495673 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:22.495207    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerStarted","Data":"b596f987d8cea2ab01b0092dcae59d4c2fe2d6bb0ae97e5ba6f52498be31300e"}
May 29 02:17:24.777088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.776984    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:24.777088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.777034    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:24.777088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.777048    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:24.777088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.777065    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:24.777088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.777077    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:24.777088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.777089    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:24.777088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.777102    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:24.777911 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.777114    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:24.779218 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.779176    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused"
May 29 02:17:24.779360 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.779222    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused"
May 29 02:17:24.779360 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.779176    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.35:8816: connect: connection refused"
May 29 02:17:24.779360 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:24.779194    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 29 02:17:27.777893 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:27.777802    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:27.778308 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:27.777919    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:27.778308 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:27.778214    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:27.778751 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:27.778729    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:27.778946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:27.778927    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:27.779068 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:27.779051    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:27.782639 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:27.782620    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:17:27.801873 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:27.801831    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podStartSLOduration=18.116484163 podStartE2EDuration="33.801820197s" podCreationTimestamp="2026-05-29 02:16:54 +0000 UTC" firstStartedPulling="2026-05-29 02:16:54.910833077 +0000 UTC m=+1111.493823960" lastFinishedPulling="2026-05-29 02:17:10.596169111 +0000 UTC m=+1127.179159994" observedRunningTime="2026-05-29 02:17:22.529116005 +0000 UTC m=+1139.112106913" watchObservedRunningTime="2026-05-29 02:17:27.801820197 +0000 UTC m=+1144.384811102"
May 29 02:17:28.517741 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:17:28.517709    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:18:23.911700 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:23.911670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:18:23.914866 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:23.914844    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:18:23.924230 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:23.924212    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:18:23.927055 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:23.927040    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:18:29.597347 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:29.597314    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"]
May 29 02:18:29.597945 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:29.597890    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="registry" containerID="cri-o://b596f987d8cea2ab01b0092dcae59d4c2fe2d6bb0ae97e5ba6f52498be31300e" gracePeriod=30
May 29 02:18:29.598071 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:29.597971    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="offline" containerID="cri-o://121ac1501890daf2ee31d0b5628d8f79ab8ebf401fe77f0f0b6b20f98f2569d4" gracePeriod=30
May 29 02:18:29.598071 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:29.598013    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="ui" containerID="cri-o://250ec128a9e675a485d2476c936f9f883f63f8c2e802740da86b2f13f92dc386" gracePeriod=30
May 29 02:18:29.598071 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:29.597958    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="online" containerID="cri-o://ad5b107ddd4ce8721ad075e2dc5a03bfd666869d66bd1129e1dcde0c18caa5e6" gracePeriod=30
May 29 02:18:29.891663 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:29.891547    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"]
May 29 02:18:29.895512 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:29.895491    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:29.897428 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:29.897406    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-dt64r\""
May 29 02:18:29.907629 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:29.907587    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"]
May 29 02:18:30.041437 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.041400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.041605 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.041454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbd54\" (UniqueName: \"kubernetes.io/projected/ccb68620-019c-4947-abd2-28377508ead9-kube-api-access-fbd54\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.041605 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.041542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.041717 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.041608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb68620-019c-4947-abd2-28377508ead9-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.041752 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.041720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.041752 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.041748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.142222 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.142117    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.142222 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.142166    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.142222 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.142195    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.142222 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.142221    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fbd54\" (UniqueName: \"kubernetes.io/projected/ccb68620-019c-4947-abd2-28377508ead9-kube-api-access-fbd54\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.142551 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.142250    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.142551 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.142290    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb68620-019c-4947-abd2-28377508ead9-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.142752 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.142730    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb68620-019c-4947-abd2-28377508ead9-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.144794 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.144766    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.144903 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.144776    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.145032 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.145013    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.145148 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.145127    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.151167 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.151145    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbd54\" (UniqueName: \"kubernetes.io/projected/ccb68620-019c-4947-abd2-28377508ead9-kube-api-access-fbd54\") pod \"feast-simple-feast-setup-664fcfcc58-nvfq4\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.205022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.204981    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:30.339337 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.339306    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"]
May 29 02:18:30.341013 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:18:30.340986    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccb68620_019c_4947_abd2_28377508ead9.slice/crio-fc4f6cb7a147adb60f0cc9393e39c439e70218da24f98ae14a6d599874c04201 WatchSource:0}: Error finding container fc4f6cb7a147adb60f0cc9393e39c439e70218da24f98ae14a6d599874c04201: Status 404 returned error can't find the container with id fc4f6cb7a147adb60f0cc9393e39c439e70218da24f98ae14a6d599874c04201
May 29 02:18:30.687569 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.687525    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerStarted","Data":"c03b674af795efbb3bbdb5c213756cad84f2e2e370a97d194368727230837bd4"}
May 29 02:18:30.687569 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.687570    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerStarted","Data":"fc4f6cb7a147adb60f0cc9393e39c439e70218da24f98ae14a6d599874c04201"}
May 29 02:18:30.690297 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.690272    2580 generic.go:358] "Generic (PLEG): container finished" podID="60d21940-92ac-4e36-918e-04ce83738a8a" containerID="250ec128a9e675a485d2476c936f9f883f63f8c2e802740da86b2f13f92dc386" exitCode=0
May 29 02:18:30.690297 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.690296    2580 generic.go:358] "Generic (PLEG): container finished" podID="60d21940-92ac-4e36-918e-04ce83738a8a" containerID="ad5b107ddd4ce8721ad075e2dc5a03bfd666869d66bd1129e1dcde0c18caa5e6" exitCode=0
May 29 02:18:30.690454 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.690302    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerDied","Data":"250ec128a9e675a485d2476c936f9f883f63f8c2e802740da86b2f13f92dc386"}
May 29 02:18:30.690454 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:30.690334    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerDied","Data":"ad5b107ddd4ce8721ad075e2dc5a03bfd666869d66bd1129e1dcde0c18caa5e6"}
May 29 02:18:35.705336 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:35.705298    2580 generic.go:358] "Generic (PLEG): container finished" podID="ccb68620-019c-4947-abd2-28377508ead9" containerID="c03b674af795efbb3bbdb5c213756cad84f2e2e370a97d194368727230837bd4" exitCode=0
May 29 02:18:35.705774 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:35.705368    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerDied","Data":"c03b674af795efbb3bbdb5c213756cad84f2e2e370a97d194368727230837bd4"}
May 29 02:18:36.709978 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:36.709947    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerStarted","Data":"f09a2e8585848b8e724011a90d40825a1a4f7b9c6675eb8be64f5be16a0849d4"}
May 29 02:18:37.778888 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:37.778843    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused"
May 29 02:18:38.515093 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:38.515051    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 29 02:18:41.729359 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:41.729325    2580 generic.go:358] "Generic (PLEG): container finished" podID="ccb68620-019c-4947-abd2-28377508ead9" containerID="f09a2e8585848b8e724011a90d40825a1a4f7b9c6675eb8be64f5be16a0849d4" exitCode=0
May 29 02:18:41.729753 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:41.729383    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerDied","Data":"f09a2e8585848b8e724011a90d40825a1a4f7b9c6675eb8be64f5be16a0849d4"}
May 29 02:18:42.737275 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:42.737233    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerStarted","Data":"bc15eb4290ac14c0134acd1c65888a0c1930ec5aa0918bf171cb696b2e294158"}
May 29 02:18:42.737824 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:42.737282    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerStarted","Data":"48479801fe600cd16a21e1ff11f5d02256d7a5a079ddbb2ae4ba875d9ab84914"}
May 29 02:18:42.737824 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:42.737296    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerStarted","Data":"5a48c8c498f66003fb0c707b89256762bfafc88d4c20c51ea04d2167cf003954"}
May 29 02:18:42.737824 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:42.737309    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerStarted","Data":"41c9b109db2a0fef74fc0b5997d7c1a97db4bdc2620b849aa41865736be900ac"}
May 29 02:18:42.761300 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:42.761236    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podStartSLOduration=13.761215009 podStartE2EDuration="13.761215009s" podCreationTimestamp="2026-05-29 02:18:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:18:42.760161201 +0000 UTC m=+1219.343152120" watchObservedRunningTime="2026-05-29 02:18:42.761215009 +0000 UTC m=+1219.344205916"
May 29 02:18:45.205794 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:45.205749    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:45.205794 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:45.205805    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:45.206344 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:45.205821    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:45.206344 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:45.206274    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:45.208009 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:45.207970    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused"
May 29 02:18:45.208009 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:45.207992    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 29 02:18:45.208213 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:45.208028    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.36:8816: connect: connection refused"
May 29 02:18:45.208213 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:45.207980    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused"
May 29 02:18:47.779121 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:47.779076    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused"
May 29 02:18:48.206724 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.206691    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.206915 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.206795    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.207035 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.207019    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.207410 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.207388    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.207532 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.207419    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.207532 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.207434    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.207654 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.207549    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.207654 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.207642    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.208022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.208003    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.211371 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.211350    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.515873 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.515777    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 29 02:18:48.757724 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.757698    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:48.760665 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:48.760647    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:18:57.778914 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:57.778859    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused"
May 29 02:18:57.779487 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:57.779038    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:18:58.515207 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:58.515157    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused"
May 29 02:18:58.515406 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:58.515291    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:18:59.793860 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:59.793827    2580 generic.go:358] "Generic (PLEG): container finished" podID="60d21940-92ac-4e36-918e-04ce83738a8a" containerID="121ac1501890daf2ee31d0b5628d8f79ab8ebf401fe77f0f0b6b20f98f2569d4" exitCode=137
May 29 02:18:59.793860 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:59.793854    2580 generic.go:358] "Generic (PLEG): container finished" podID="60d21940-92ac-4e36-918e-04ce83738a8a" containerID="b596f987d8cea2ab01b0092dcae59d4c2fe2d6bb0ae97e5ba6f52498be31300e" exitCode=137
May 29 02:18:59.794256 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:59.793895    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerDied","Data":"121ac1501890daf2ee31d0b5628d8f79ab8ebf401fe77f0f0b6b20f98f2569d4"}
May 29 02:18:59.794256 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:18:59.793931    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerDied","Data":"b596f987d8cea2ab01b0092dcae59d4c2fe2d6bb0ae97e5ba6f52498be31300e"}
May 29 02:19:00.250107 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.250085    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:19:00.390439 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.390352    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfzpf\" (UniqueName: \"kubernetes.io/projected/60d21940-92ac-4e36-918e-04ce83738a8a-kube-api-access-dfzpf\") pod \"60d21940-92ac-4e36-918e-04ce83738a8a\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") "
May 29 02:19:00.390439 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.390396    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60d21940-92ac-4e36-918e-04ce83738a8a-feast-data\") pod \"60d21940-92ac-4e36-918e-04ce83738a8a\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") "
May 29 02:19:00.390439 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.390435    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-offline-tls\") pod \"60d21940-92ac-4e36-918e-04ce83738a8a\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") "
May 29 02:19:00.390755 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.390501    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-online-tls\") pod \"60d21940-92ac-4e36-918e-04ce83738a8a\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") "
May 29 02:19:00.390755 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.390523    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-registry-tls\") pod \"60d21940-92ac-4e36-918e-04ce83738a8a\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") "
May 29 02:19:00.390755 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.390543    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-ui-tls\") pod \"60d21940-92ac-4e36-918e-04ce83738a8a\" (UID: \"60d21940-92ac-4e36-918e-04ce83738a8a\") "
May 29 02:19:00.391113 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.391068    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d21940-92ac-4e36-918e-04ce83738a8a-feast-data" (OuterVolumeSpecName: "feast-data") pod "60d21940-92ac-4e36-918e-04ce83738a8a" (UID: "60d21940-92ac-4e36-918e-04ce83738a8a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:19:00.393118 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.393084    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "60d21940-92ac-4e36-918e-04ce83738a8a" (UID: "60d21940-92ac-4e36-918e-04ce83738a8a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:19:00.393118 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.393100    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-online-tls" (OuterVolumeSpecName: "online-tls") pod "60d21940-92ac-4e36-918e-04ce83738a8a" (UID: "60d21940-92ac-4e36-918e-04ce83738a8a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:19:00.393276 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.393174    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "60d21940-92ac-4e36-918e-04ce83738a8a" (UID: "60d21940-92ac-4e36-918e-04ce83738a8a"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:19:00.393276 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.393192    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d21940-92ac-4e36-918e-04ce83738a8a-kube-api-access-dfzpf" (OuterVolumeSpecName: "kube-api-access-dfzpf") pod "60d21940-92ac-4e36-918e-04ce83738a8a" (UID: "60d21940-92ac-4e36-918e-04ce83738a8a"). InnerVolumeSpecName "kube-api-access-dfzpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:19:00.393276 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.393241    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "60d21940-92ac-4e36-918e-04ce83738a8a" (UID: "60d21940-92ac-4e36-918e-04ce83738a8a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:19:00.492110 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.492072    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfzpf\" (UniqueName: \"kubernetes.io/projected/60d21940-92ac-4e36-918e-04ce83738a8a-kube-api-access-dfzpf\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:19:00.492110 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.492104    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/60d21940-92ac-4e36-918e-04ce83738a8a-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:19:00.492110 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.492114    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:19:00.492340 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.492141    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:19:00.492340 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.492149    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:19:00.492340 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.492157    2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/60d21940-92ac-4e36-918e-04ce83738a8a-ui-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:19:00.799828 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.799799    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v" event={"ID":"60d21940-92ac-4e36-918e-04ce83738a8a","Type":"ContainerDied","Data":"820adad9c84966ca5dabec73024f806355e6bd902272dc79c250c3d43bfc00ed"}
May 29 02:19:00.799828 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.799819    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"
May 29 02:19:00.800311 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.799848    2580 scope.go:117] "RemoveContainer" containerID="250ec128a9e675a485d2476c936f9f883f63f8c2e802740da86b2f13f92dc386"
May 29 02:19:00.809475 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.809454    2580 scope.go:117] "RemoveContainer" containerID="121ac1501890daf2ee31d0b5628d8f79ab8ebf401fe77f0f0b6b20f98f2569d4"
May 29 02:19:00.818718 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.818700    2580 scope.go:117] "RemoveContainer" containerID="ad5b107ddd4ce8721ad075e2dc5a03bfd666869d66bd1129e1dcde0c18caa5e6"
May 29 02:19:00.822411 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.822390    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"]
May 29 02:19:00.826539 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.826522    2580 scope.go:117] "RemoveContainer" containerID="b596f987d8cea2ab01b0092dcae59d4c2fe2d6bb0ae97e5ba6f52498be31300e"
May 29 02:19:00.827555 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.827540    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-qlj7v"]
May 29 02:19:00.833403 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.833385    2580 scope.go:117] "RemoveContainer" containerID="f5496c5313a7ba849e1f2fd0e7bbbd46811732e5d51c35c04ecf60f839ff2b26"
May 29 02:19:00.839993 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:00.839978    2580 scope.go:117] "RemoveContainer" containerID="5ff7c2921458e78b4d4626a5534aacc4fdba321dafbdda4ee0a663f16e5daf52"
May 29 02:19:01.974039 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:01.974005    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" path="/var/lib/kubelet/pods/60d21940-92ac-4e36-918e-04ce83738a8a/volumes"
May 29 02:19:51.668177 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668141    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"]
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668425    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="feast-apply"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668439    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="feast-apply"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668454    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="online"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668462    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="online"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668473    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="ui"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668480    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="ui"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668491    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="feast-init"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668500    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="feast-init"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668507    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="offline"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668511    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="offline"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668520    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="registry"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668525    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="registry"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668619    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="online"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668633    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="registry"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668640    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="ui"
May 29 02:19:51.670781 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.668649    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="60d21940-92ac-4e36-918e-04ce83738a8a" containerName="offline"
May 29 02:19:51.672388 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.672372    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.674516 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.674492    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 29 02:19:51.674645 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.674515    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 29 02:19:51.674765 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.674726    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 29 02:19:51.674857 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.674808    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 29 02:19:51.674857 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.674824    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-d7rzr\""
May 29 02:19:51.674967 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.674869    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 29 02:19:51.675081 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.675064    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 29 02:19:51.683270 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.683249    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"]
May 29 02:19:51.777398 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.777360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.777581 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.777481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvt7j\" (UniqueName: \"kubernetes.io/projected/89860242-2117-41aa-b4fe-5dcdf1cd456e-kube-api-access-tvt7j\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.777581 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.777510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.777581 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.777530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89860242-2117-41aa-b4fe-5dcdf1cd456e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.777581 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.777553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/89860242-2117-41aa-b4fe-5dcdf1cd456e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.777581 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.777570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.878895 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.878807    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tvt7j\" (UniqueName: \"kubernetes.io/projected/89860242-2117-41aa-b4fe-5dcdf1cd456e-kube-api-access-tvt7j\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.879077 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.878922    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.879077 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.878942    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89860242-2117-41aa-b4fe-5dcdf1cd456e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.879077 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.878963    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/89860242-2117-41aa-b4fe-5dcdf1cd456e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.879077 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.878979    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.879077 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.879010    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.879405 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.879379    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89860242-2117-41aa-b4fe-5dcdf1cd456e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.879707 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.879683    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/89860242-2117-41aa-b4fe-5dcdf1cd456e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.881500 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.881470    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.881649 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.881536    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.881649 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.881637    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.886784 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.886763    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvt7j\" (UniqueName: \"kubernetes.io/projected/89860242-2117-41aa-b4fe-5dcdf1cd456e-kube-api-access-tvt7j\") pod \"feast-simple-feast-remote-setup-5cff94746b-k4c8g\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:51.981564 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:51.981479    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:19:52.115094 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:52.115069    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"]
May 29 02:19:52.116954 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:19:52.116931    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89860242_2117_41aa_b4fe_5dcdf1cd456e.slice/crio-53679f7d9f504a895a98bf3a8b682ee8fcc0ae2ef56458880362a11795ac178c WatchSource:0}: Error finding container 53679f7d9f504a895a98bf3a8b682ee8fcc0ae2ef56458880362a11795ac178c: Status 404 returned error can't find the container with id 53679f7d9f504a895a98bf3a8b682ee8fcc0ae2ef56458880362a11795ac178c
May 29 02:19:52.946859 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:52.946823    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerStarted","Data":"a6e9c2f152528c2913b5e11459d71a92a3a42dd8192b3a8cb9b1088a0dc991c5"}
May 29 02:19:52.946859 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:52.946859    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerStarted","Data":"53679f7d9f504a895a98bf3a8b682ee8fcc0ae2ef56458880362a11795ac178c"}
May 29 02:19:56.959457 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:56.959353    2580 generic.go:358] "Generic (PLEG): container finished" podID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerID="a6e9c2f152528c2913b5e11459d71a92a3a42dd8192b3a8cb9b1088a0dc991c5" exitCode=0
May 29 02:19:56.959457 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:56.959397    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerDied","Data":"a6e9c2f152528c2913b5e11459d71a92a3a42dd8192b3a8cb9b1088a0dc991c5"}
May 29 02:19:57.964061 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:19:57.964026    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerStarted","Data":"7a76ac9d64d3ccba5909722f9ef4a18bc4fd4afc6f01dbad4f1ee0f414afcd7f"}
May 29 02:20:02.979317 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:02.979232    2580 generic.go:358] "Generic (PLEG): container finished" podID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerID="7a76ac9d64d3ccba5909722f9ef4a18bc4fd4afc6f01dbad4f1ee0f414afcd7f" exitCode=0
May 29 02:20:02.979697 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:02.979307    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerDied","Data":"7a76ac9d64d3ccba5909722f9ef4a18bc4fd4afc6f01dbad4f1ee0f414afcd7f"}
May 29 02:20:03.985153 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:03.985120    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerStarted","Data":"8a0c4e7a01338580c58bab726c38b5b47c44ad75567641d2e53207b6def5d2b7"}
May 29 02:20:03.985631 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:03.985164    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerStarted","Data":"0dc8894078706ef236ba3ecd4c979f975fa0874ba1714b27d8246f2d0da6d4fc"}
May 29 02:20:03.985631 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:03.985177    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerStarted","Data":"6c7a6eaec0f0a4b2c6b298806d8b3b447b27e3ce9bf006f2f14aa85bad909fb0"}
May 29 02:20:04.009358 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:04.009296    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podStartSLOduration=13.009278651 podStartE2EDuration="13.009278651s" podCreationTimestamp="2026-05-29 02:19:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:20:04.008022997 +0000 UTC m=+1300.591013902" watchObservedRunningTime="2026-05-29 02:20:04.009278651 +0000 UTC m=+1300.592269557"
May 29 02:20:06.982263 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:06.982210    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:06.982263 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:06.982268    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:06.982709 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:06.982285    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:06.984211 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:06.984174    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.37:8443: connect: connection refused"
May 29 02:20:06.984315 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:06.984221    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused"
May 29 02:20:06.984448 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:06.984390    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.37:8816: connect: connection refused"
May 29 02:20:09.982705 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:09.982678    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:09.983167 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:09.982785    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:09.983249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:09.983204    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:09.983587 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:09.983569    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:09.983690 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:09.983661    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:09.983727 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:09.983703    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:09.986825 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:09.986805    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:10.006663 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:10.006642    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:20:10.009434 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:20:10.009419    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:21:12.859788 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:12.859752    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"]
May 29 02:21:12.860369 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:12.860163    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="online" containerID="cri-o://6c7a6eaec0f0a4b2c6b298806d8b3b447b27e3ce9bf006f2f14aa85bad909fb0" gracePeriod=30
May 29 02:21:12.869621 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:12.863006    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="offline" containerID="cri-o://0dc8894078706ef236ba3ecd4c979f975fa0874ba1714b27d8246f2d0da6d4fc" gracePeriod=30
May 29 02:21:12.870288 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:12.860766    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="ui" containerID="cri-o://8a0c4e7a01338580c58bab726c38b5b47c44ad75567641d2e53207b6def5d2b7" gracePeriod=30
May 29 02:21:13.009948 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:13.009910    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"]
May 29 02:21:13.010385 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:13.010351    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="registry" containerID="cri-o://41c9b109db2a0fef74fc0b5997d7c1a97db4bdc2620b849aa41865736be900ac" gracePeriod=30
May 29 02:21:13.010608 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:13.010483    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="offline" containerID="cri-o://48479801fe600cd16a21e1ff11f5d02256d7a5a079ddbb2ae4ba875d9ab84914" gracePeriod=30
May 29 02:21:13.010608 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:13.010521    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="online" containerID="cri-o://5a48c8c498f66003fb0c707b89256762bfafc88d4c20c51ea04d2167cf003954" gracePeriod=30
May 29 02:21:13.010820 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:13.010580    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="ui" containerID="cri-o://bc15eb4290ac14c0134acd1c65888a0c1930ec5aa0918bf171cb696b2e294158" gracePeriod=30
May 29 02:21:14.189414 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:14.189380    2580 generic.go:358] "Generic (PLEG): container finished" podID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerID="8a0c4e7a01338580c58bab726c38b5b47c44ad75567641d2e53207b6def5d2b7" exitCode=0
May 29 02:21:14.189414 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:14.189405    2580 generic.go:358] "Generic (PLEG): container finished" podID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerID="6c7a6eaec0f0a4b2c6b298806d8b3b447b27e3ce9bf006f2f14aa85bad909fb0" exitCode=0
May 29 02:21:14.189891 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:14.189454    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerDied","Data":"8a0c4e7a01338580c58bab726c38b5b47c44ad75567641d2e53207b6def5d2b7"}
May 29 02:21:14.189891 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:14.189491    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerDied","Data":"6c7a6eaec0f0a4b2c6b298806d8b3b447b27e3ce9bf006f2f14aa85bad909fb0"}
May 29 02:21:14.191744 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:14.191721    2580 generic.go:358] "Generic (PLEG): container finished" podID="ccb68620-019c-4947-abd2-28377508ead9" containerID="bc15eb4290ac14c0134acd1c65888a0c1930ec5aa0918bf171cb696b2e294158" exitCode=0
May 29 02:21:14.191744 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:14.191742    2580 generic.go:358] "Generic (PLEG): container finished" podID="ccb68620-019c-4947-abd2-28377508ead9" containerID="5a48c8c498f66003fb0c707b89256762bfafc88d4c20c51ea04d2167cf003954" exitCode=0
May 29 02:21:14.191904 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:14.191777    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerDied","Data":"bc15eb4290ac14c0134acd1c65888a0c1930ec5aa0918bf171cb696b2e294158"}
May 29 02:21:14.191904 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:14.191796    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerDied","Data":"5a48c8c498f66003fb0c707b89256762bfafc88d4c20c51ea04d2167cf003954"}
May 29 02:21:18.208455 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:18.208409    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused"
May 29 02:21:18.758136 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:18.758091    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 29 02:21:19.984103 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:19.984054    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.37:8443: connect: connection refused"
May 29 02:21:20.006783 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:20.006737    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused"
May 29 02:21:28.208203 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:28.208157    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused"
May 29 02:21:28.757998 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:28.757953    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 29 02:21:29.984128 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:29.984086    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.37:8443: connect: connection refused"
May 29 02:21:30.007377 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:30.007348    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused"
May 29 02:21:38.207746 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:38.207704    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused"
May 29 02:21:38.208175 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:38.207859    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:21:38.758563 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:38.758520    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused"
May 29 02:21:38.758761 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:38.758688    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:21:39.983657 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:39.983613    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.37:8443: connect: connection refused"
May 29 02:21:39.984098 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:39.983753    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:21:40.007073 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:40.007037    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.37:6567/health\": dial tcp 10.134.0.37:6567: connect: connection refused"
May 29 02:21:40.007204 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:40.007138    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:21:43.278698 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.278580    2580 generic.go:358] "Generic (PLEG): container finished" podID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerID="0dc8894078706ef236ba3ecd4c979f975fa0874ba1714b27d8246f2d0da6d4fc" exitCode=137
May 29 02:21:43.278698 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.278622    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerDied","Data":"0dc8894078706ef236ba3ecd4c979f975fa0874ba1714b27d8246f2d0da6d4fc"}
May 29 02:21:43.281143 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.281119    2580 generic.go:358] "Generic (PLEG): container finished" podID="ccb68620-019c-4947-abd2-28377508ead9" containerID="48479801fe600cd16a21e1ff11f5d02256d7a5a079ddbb2ae4ba875d9ab84914" exitCode=137
May 29 02:21:43.281143 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.281138    2580 generic.go:358] "Generic (PLEG): container finished" podID="ccb68620-019c-4947-abd2-28377508ead9" containerID="41c9b109db2a0fef74fc0b5997d7c1a97db4bdc2620b849aa41865736be900ac" exitCode=137
May 29 02:21:43.281304 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.281189    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerDied","Data":"48479801fe600cd16a21e1ff11f5d02256d7a5a079ddbb2ae4ba875d9ab84914"}
May 29 02:21:43.281304 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.281216    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerDied","Data":"41c9b109db2a0fef74fc0b5997d7c1a97db4bdc2620b849aa41865736be900ac"}
May 29 02:21:43.512513 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.512491    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:21:43.576621 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.576572    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89860242-2117-41aa-b4fe-5dcdf1cd456e-feast-data\") pod \"89860242-2117-41aa-b4fe-5dcdf1cd456e\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") "
May 29 02:21:43.576804 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.576658    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-offline-tls\") pod \"89860242-2117-41aa-b4fe-5dcdf1cd456e\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") "
May 29 02:21:43.576804 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.576687    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/89860242-2117-41aa-b4fe-5dcdf1cd456e-registry-tls\") pod \"89860242-2117-41aa-b4fe-5dcdf1cd456e\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") "
May 29 02:21:43.576804 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.576727    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-online-tls\") pod \"89860242-2117-41aa-b4fe-5dcdf1cd456e\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") "
May 29 02:21:43.576969 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.576813    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-ui-tls\") pod \"89860242-2117-41aa-b4fe-5dcdf1cd456e\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") "
May 29 02:21:43.576969 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.576840    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvt7j\" (UniqueName: \"kubernetes.io/projected/89860242-2117-41aa-b4fe-5dcdf1cd456e-kube-api-access-tvt7j\") pod \"89860242-2117-41aa-b4fe-5dcdf1cd456e\" (UID: \"89860242-2117-41aa-b4fe-5dcdf1cd456e\") "
May 29 02:21:43.577152 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.577067    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89860242-2117-41aa-b4fe-5dcdf1cd456e-feast-data" (OuterVolumeSpecName: "feast-data") pod "89860242-2117-41aa-b4fe-5dcdf1cd456e" (UID: "89860242-2117-41aa-b4fe-5dcdf1cd456e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:21:43.578025 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.577887    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89860242-2117-41aa-b4fe-5dcdf1cd456e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "89860242-2117-41aa-b4fe-5dcdf1cd456e" (UID: "89860242-2117-41aa-b4fe-5dcdf1cd456e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:21:43.581642 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.581613    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89860242-2117-41aa-b4fe-5dcdf1cd456e-kube-api-access-tvt7j" (OuterVolumeSpecName: "kube-api-access-tvt7j") pod "89860242-2117-41aa-b4fe-5dcdf1cd456e" (UID: "89860242-2117-41aa-b4fe-5dcdf1cd456e"). InnerVolumeSpecName "kube-api-access-tvt7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:21:43.584285 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.584252    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "89860242-2117-41aa-b4fe-5dcdf1cd456e" (UID: "89860242-2117-41aa-b4fe-5dcdf1cd456e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:21:43.584385 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.584319    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-online-tls" (OuterVolumeSpecName: "online-tls") pod "89860242-2117-41aa-b4fe-5dcdf1cd456e" (UID: "89860242-2117-41aa-b4fe-5dcdf1cd456e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:21:43.584385 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.584347    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "89860242-2117-41aa-b4fe-5dcdf1cd456e" (UID: "89860242-2117-41aa-b4fe-5dcdf1cd456e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:21:43.654485 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.654461    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:21:43.677426 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677387    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb68620-019c-4947-abd2-28377508ead9-feast-data\") pod \"ccb68620-019c-4947-abd2-28377508ead9\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") "
May 29 02:21:43.677581 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677458    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-offline-tls\") pod \"ccb68620-019c-4947-abd2-28377508ead9\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") "
May 29 02:21:43.677581 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677494    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-online-tls\") pod \"ccb68620-019c-4947-abd2-28377508ead9\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") "
May 29 02:21:43.677581 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677526    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbd54\" (UniqueName: \"kubernetes.io/projected/ccb68620-019c-4947-abd2-28377508ead9-kube-api-access-fbd54\") pod \"ccb68620-019c-4947-abd2-28377508ead9\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") "
May 29 02:21:43.677581 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677567    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-registry-tls\") pod \"ccb68620-019c-4947-abd2-28377508ead9\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") "
May 29 02:21:43.677824 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677622    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-ui-tls\") pod \"ccb68620-019c-4947-abd2-28377508ead9\" (UID: \"ccb68620-019c-4947-abd2-28377508ead9\") "
May 29 02:21:43.677879 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677850    2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-ui-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.677879 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677868    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvt7j\" (UniqueName: \"kubernetes.io/projected/89860242-2117-41aa-b4fe-5dcdf1cd456e-kube-api-access-tvt7j\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.678032 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677884    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89860242-2117-41aa-b4fe-5dcdf1cd456e-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.678032 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677897    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.678032 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677909    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/89860242-2117-41aa-b4fe-5dcdf1cd456e-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.678032 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677919    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89860242-2117-41aa-b4fe-5dcdf1cd456e-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.678032 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.677978    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb68620-019c-4947-abd2-28377508ead9-feast-data" (OuterVolumeSpecName: "feast-data") pod "ccb68620-019c-4947-abd2-28377508ead9" (UID: "ccb68620-019c-4947-abd2-28377508ead9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:21:43.680152 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.680118    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-online-tls" (OuterVolumeSpecName: "online-tls") pod "ccb68620-019c-4947-abd2-28377508ead9" (UID: "ccb68620-019c-4947-abd2-28377508ead9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:21:43.680274 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.680202    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb68620-019c-4947-abd2-28377508ead9-kube-api-access-fbd54" (OuterVolumeSpecName: "kube-api-access-fbd54") pod "ccb68620-019c-4947-abd2-28377508ead9" (UID: "ccb68620-019c-4947-abd2-28377508ead9"). InnerVolumeSpecName "kube-api-access-fbd54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:21:43.680336 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.680298    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "ccb68620-019c-4947-abd2-28377508ead9" (UID: "ccb68620-019c-4947-abd2-28377508ead9"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:21:43.680415 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.680400    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ccb68620-019c-4947-abd2-28377508ead9" (UID: "ccb68620-019c-4947-abd2-28377508ead9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:21:43.680866 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.680842    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ccb68620-019c-4947-abd2-28377508ead9" (UID: "ccb68620-019c-4947-abd2-28377508ead9"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:21:43.778925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.778879    2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-ui-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.778925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.778919    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ccb68620-019c-4947-abd2-28377508ead9-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.778925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.778929    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.778925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.778938    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.779216 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.778946    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbd54\" (UniqueName: \"kubernetes.io/projected/ccb68620-019c-4947-abd2-28377508ead9-kube-api-access-fbd54\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:43.779216 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:43.778955    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ccb68620-019c-4947-abd2-28377508ead9-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:21:44.286158 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.286074    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g" event={"ID":"89860242-2117-41aa-b4fe-5dcdf1cd456e","Type":"ContainerDied","Data":"53679f7d9f504a895a98bf3a8b682ee8fcc0ae2ef56458880362a11795ac178c"}
May 29 02:21:44.286158 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.286121    2580 scope.go:117] "RemoveContainer" containerID="8a0c4e7a01338580c58bab726c38b5b47c44ad75567641d2e53207b6def5d2b7"
May 29 02:21:44.286658 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.286087    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"
May 29 02:21:44.291477 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.290220    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4" event={"ID":"ccb68620-019c-4947-abd2-28377508ead9","Type":"ContainerDied","Data":"fc4f6cb7a147adb60f0cc9393e39c439e70218da24f98ae14a6d599874c04201"}
May 29 02:21:44.291477 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.290587    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"
May 29 02:21:44.300633 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.300563    2580 scope.go:117] "RemoveContainer" containerID="0dc8894078706ef236ba3ecd4c979f975fa0874ba1714b27d8246f2d0da6d4fc"
May 29 02:21:44.308114 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.308092    2580 scope.go:117] "RemoveContainer" containerID="6c7a6eaec0f0a4b2c6b298806d8b3b447b27e3ce9bf006f2f14aa85bad909fb0"
May 29 02:21:44.310948 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.310925    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"]
May 29 02:21:44.315443 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.315418    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-k4c8g"]
May 29 02:21:44.315982 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.315967    2580 scope.go:117] "RemoveContainer" containerID="7a76ac9d64d3ccba5909722f9ef4a18bc4fd4afc6f01dbad4f1ee0f414afcd7f"
May 29 02:21:44.322894 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.322880    2580 scope.go:117] "RemoveContainer" containerID="a6e9c2f152528c2913b5e11459d71a92a3a42dd8192b3a8cb9b1088a0dc991c5"
May 29 02:21:44.326200 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.326176    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"]
May 29 02:21:44.329773 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.329754    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-nvfq4"]
May 29 02:21:44.330117 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.330100    2580 scope.go:117] "RemoveContainer" containerID="bc15eb4290ac14c0134acd1c65888a0c1930ec5aa0918bf171cb696b2e294158"
May 29 02:21:44.336989 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.336969    2580 scope.go:117] "RemoveContainer" containerID="48479801fe600cd16a21e1ff11f5d02256d7a5a079ddbb2ae4ba875d9ab84914"
May 29 02:21:44.343233 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.343216    2580 scope.go:117] "RemoveContainer" containerID="5a48c8c498f66003fb0c707b89256762bfafc88d4c20c51ea04d2167cf003954"
May 29 02:21:44.349584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.349568    2580 scope.go:117] "RemoveContainer" containerID="41c9b109db2a0fef74fc0b5997d7c1a97db4bdc2620b849aa41865736be900ac"
May 29 02:21:44.355651 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.355635    2580 scope.go:117] "RemoveContainer" containerID="f09a2e8585848b8e724011a90d40825a1a4f7b9c6675eb8be64f5be16a0849d4"
May 29 02:21:44.361948 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:44.361932    2580 scope.go:117] "RemoveContainer" containerID="c03b674af795efbb3bbdb5c213756cad84f2e2e370a97d194368727230837bd4"
May 29 02:21:45.974178 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:45.974143    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" path="/var/lib/kubelet/pods/89860242-2117-41aa-b4fe-5dcdf1cd456e/volumes"
May 29 02:21:45.974705 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:21:45.974692    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb68620-019c-4947-abd2-28377508ead9" path="/var/lib/kubelet/pods/ccb68620-019c-4947-abd2-28377508ead9/volumes"
May 29 02:22:25.525471 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525436    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"]
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525734    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="feast-init"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525746    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="feast-init"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525758    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="online"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525764    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="online"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525773    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="ui"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525779    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="ui"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525790    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="registry"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525798    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="registry"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525810    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="online"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525815    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="online"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525821    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="offline"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525826    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="offline"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525833    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="ui"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525838    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="ui"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525843    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="feast-apply"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525848    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="feast-apply"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525854    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="offline"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525858    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="offline"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525865    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="feast-init"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525873    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="feast-init"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525884    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="feast-apply"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525889    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="feast-apply"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525932    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="offline"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525941    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="online"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525951    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="89860242-2117-41aa-b4fe-5dcdf1cd456e" containerName="ui"
May 29 02:22:25.525946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525962    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="offline"
May 29 02:22:25.526864 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525971    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="registry"
May 29 02:22:25.526864 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525981    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="online"
May 29 02:22:25.526864 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.525987    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ccb68620-019c-4947-abd2-28377508ead9" containerName="ui"
May 29 02:22:25.530925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.530909    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.533377 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.533353    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 29 02:22:25.533377 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.533358    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 29 02:22:25.533549 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.533358    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 29 02:22:25.533828 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.533809    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 29 02:22:25.533920 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.533832    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-b5k9t\""
May 29 02:22:25.533980 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.533967    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 29 02:22:25.534234 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.534217    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 29 02:22:25.543637 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.543616    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"]
May 29 02:22:25.694116 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.694084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.694116 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.694121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.694327 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.694210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.694327 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.694235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.694327 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.694262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdba8be-61e8-4188-9657-4a5fca2e722b-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.694327 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.694296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zw4t\" (UniqueName: \"kubernetes.io/projected/ecdba8be-61e8-4188-9657-4a5fca2e722b-kube-api-access-7zw4t\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.794661 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.794634    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.794816 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.794669    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.794816 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.794685    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdba8be-61e8-4188-9657-4a5fca2e722b-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.794914 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.794819    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7zw4t\" (UniqueName: \"kubernetes.io/projected/ecdba8be-61e8-4188-9657-4a5fca2e722b-kube-api-access-7zw4t\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.794914 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.794865    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.794914 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.794899    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.795090 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.795070    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdba8be-61e8-4188-9657-4a5fca2e722b-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.797297 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.797269    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.797297 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.797277    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.797297 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.797294    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.797556 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.797277    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.804698 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.804670    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zw4t\" (UniqueName: \"kubernetes.io/projected/ecdba8be-61e8-4188-9657-4a5fca2e722b-kube-api-access-7zw4t\") pod \"feast-simple-feast-setup-664fcfcc58-n8mwh\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.859664 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.859630    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:25.987038 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.987011    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"]
May 29 02:22:25.990227 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:22:25.990202    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecdba8be_61e8_4188_9657_4a5fca2e722b.slice/crio-a5a468151ceef40ad68eec22f433b23b1348cddb448a309da0a2c0d787d1e55c WatchSource:0}: Error finding container a5a468151ceef40ad68eec22f433b23b1348cddb448a309da0a2c0d787d1e55c: Status 404 returned error can't find the container with id a5a468151ceef40ad68eec22f433b23b1348cddb448a309da0a2c0d787d1e55c
May 29 02:22:25.991940 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:25.991926    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 02:22:26.412340 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:26.412253    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerStarted","Data":"adac5c3ecf43c35905a95c025a401c6d2d570c9c13b3bfd9df00e1da21f44c82"}
May 29 02:22:26.412340 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:26.412290    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerStarted","Data":"a5a468151ceef40ad68eec22f433b23b1348cddb448a309da0a2c0d787d1e55c"}
May 29 02:22:31.426644 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:31.426609    2580 generic.go:358] "Generic (PLEG): container finished" podID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerID="adac5c3ecf43c35905a95c025a401c6d2d570c9c13b3bfd9df00e1da21f44c82" exitCode=0
May 29 02:22:31.427159 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:31.426683    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerDied","Data":"adac5c3ecf43c35905a95c025a401c6d2d570c9c13b3bfd9df00e1da21f44c82"}
May 29 02:22:32.431413 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:32.431380    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerStarted","Data":"f79b1073dfb60c8e22ab20e5b3f83b60b9f62b0120a44a2e55c45ad02d9b821d"}
May 29 02:22:37.450772 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:37.450731    2580 generic.go:358] "Generic (PLEG): container finished" podID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerID="f79b1073dfb60c8e22ab20e5b3f83b60b9f62b0120a44a2e55c45ad02d9b821d" exitCode=0
May 29 02:22:37.451163 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:37.450803    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerDied","Data":"f79b1073dfb60c8e22ab20e5b3f83b60b9f62b0120a44a2e55c45ad02d9b821d"}
May 29 02:22:38.458930 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:38.458888    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerStarted","Data":"869b13ddb431b051bcaac6302501ff5ccc2412d28c8397683f7a978f02716d73"}
May 29 02:22:38.459497 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:38.458942    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerStarted","Data":"070e553f644f21da0fbb0bdcd851d0741cb83345712804166138f3d77d9ccb4e"}
May 29 02:22:38.459497 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:38.458956    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerStarted","Data":"06820b2458144f304b7d029f6bc83e0bf2066b50d93d76ef3e4b459d355bf342"}
May 29 02:22:38.459497 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:38.458968    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerStarted","Data":"e0236869988bb76c38a677d2b6c0cb52d29f03d8bdf5628b0eef63d618728563"}
May 29 02:22:38.484322 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:38.484265    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podStartSLOduration=13.484246634 podStartE2EDuration="13.484246634s" podCreationTimestamp="2026-05-29 02:22:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:22:38.482271391 +0000 UTC m=+1455.065262297" watchObservedRunningTime="2026-05-29 02:22:38.484246634 +0000 UTC m=+1455.067237540"
May 29 02:22:40.861646 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:40.860697    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:40.861646 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:40.860750    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:40.861646 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:40.860766    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:40.861646 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:40.860777    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:40.862995 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:40.862954    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused"
May 29 02:22:40.863136 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:40.862986    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.38:8816: connect: connection refused"
May 29 02:22:40.863136 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:40.863035    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.38:6571: connect: connection refused"
May 29 02:22:40.863136 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:40.862997    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused"
May 29 02:22:43.861436 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.861400    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:43.861930 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.861650    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:43.861930 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.861762    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:43.862125 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.862101    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:43.862249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.862134    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:43.862249 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.862147    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:43.862384 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.862256    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:43.862425 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.862401    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:43.862565 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.862551    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:43.866088 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:43.866073    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:44.484681 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:44.484653    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:22:44.487643 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:22:44.487619    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:23:23.931409 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:23.931380    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:23:23.934511 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:23.934492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:23:23.947055 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:23.947035    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:23:23.949923 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:23.949905    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:23:46.300187 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.300107    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"]
May 29 02:23:46.300633 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.300556    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="registry" containerID="cri-o://e0236869988bb76c38a677d2b6c0cb52d29f03d8bdf5628b0eef63d618728563" gracePeriod=30
May 29 02:23:46.300796 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.300628    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="offline" containerID="cri-o://070e553f644f21da0fbb0bdcd851d0741cb83345712804166138f3d77d9ccb4e" gracePeriod=30
May 29 02:23:46.300796 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.300633    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="online" containerID="cri-o://06820b2458144f304b7d029f6bc83e0bf2066b50d93d76ef3e4b459d355bf342" gracePeriod=30
May 29 02:23:46.300796 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.300734    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="ui" containerID="cri-o://869b13ddb431b051bcaac6302501ff5ccc2412d28c8397683f7a978f02716d73" gracePeriod=30
May 29 02:23:46.688532 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.688434    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"]
May 29 02:23:46.692363 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.692340    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.694455 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.694432    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-9vwrv\""
May 29 02:23:46.703523 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.703502    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"]
May 29 02:23:46.825252 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.825222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4e252561-bcdc-4edc-88e5-d40b05720cb1-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.825391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.825268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.825391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.825290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.825391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.825363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.825505 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.825396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnnlz\" (UniqueName: \"kubernetes.io/projected/4e252561-bcdc-4edc-88e5-d40b05720cb1-kube-api-access-nnnlz\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.825505 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.825462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.926248 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.926210    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nnnlz\" (UniqueName: \"kubernetes.io/projected/4e252561-bcdc-4edc-88e5-d40b05720cb1-kube-api-access-nnnlz\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.926442 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.926255    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.926442 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.926293    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4e252561-bcdc-4edc-88e5-d40b05720cb1-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.926442 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.926323    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.926442 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.926350    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.926442 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.926394    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.926743 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:23:46.926455    2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 29 02:23:46.926743 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:23:46.926494    2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 29 02:23:46.926743 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:23:46.926538    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-online-tls podName:4e252561-bcdc-4edc-88e5-d40b05720cb1 nodeName:}" failed. No retries permitted until 2026-05-29 02:23:47.42651762 +0000 UTC m=+1524.009508518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-online-tls") pod "feast-simple-feast-setup-664fcfcc58-54wz9" (UID: "4e252561-bcdc-4edc-88e5-d40b05720cb1") : secret "feast-simple-feast-setup-online-tls" not found
May 29 02:23:46.926743 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:23:46.926557    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-offline-tls podName:4e252561-bcdc-4edc-88e5-d40b05720cb1 nodeName:}" failed. No retries permitted until 2026-05-29 02:23:47.426548397 +0000 UTC m=+1524.009539282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-54wz9" (UID: "4e252561-bcdc-4edc-88e5-d40b05720cb1") : secret "feast-simple-feast-setup-offline-tls" not found
May 29 02:23:46.926920 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.926760    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4e252561-bcdc-4edc-88e5-d40b05720cb1-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.928783 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.928760    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.928879 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.928857    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:46.934287 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:46.934265    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnnlz\" (UniqueName: \"kubernetes.io/projected/4e252561-bcdc-4edc-88e5-d40b05720cb1-kube-api-access-nnnlz\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:47.430159 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.430121    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:47.430159 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.430161    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:47.432575 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.432550    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:47.432701 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.432620    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-54wz9\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:47.601405 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.601353    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:23:47.659690 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.659455    2580 generic.go:358] "Generic (PLEG): container finished" podID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerID="869b13ddb431b051bcaac6302501ff5ccc2412d28c8397683f7a978f02716d73" exitCode=0
May 29 02:23:47.659690 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.659488    2580 generic.go:358] "Generic (PLEG): container finished" podID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerID="06820b2458144f304b7d029f6bc83e0bf2066b50d93d76ef3e4b459d355bf342" exitCode=0
May 29 02:23:47.659690 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.659528    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerDied","Data":"869b13ddb431b051bcaac6302501ff5ccc2412d28c8397683f7a978f02716d73"}
May 29 02:23:47.659690 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.659572    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerDied","Data":"06820b2458144f304b7d029f6bc83e0bf2066b50d93d76ef3e4b459d355bf342"}
May 29 02:23:47.738538 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:47.738502    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"]
May 29 02:23:47.739531 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:23:47.739501    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e252561_bcdc_4edc_88e5_d40b05720cb1.slice/crio-66ac87b50e191b27ef4f82dcb0c96adadef5c14b059d7149d84a5119ffe14aba WatchSource:0}: Error finding container 66ac87b50e191b27ef4f82dcb0c96adadef5c14b059d7149d84a5119ffe14aba: Status 404 returned error can't find the container with id 66ac87b50e191b27ef4f82dcb0c96adadef5c14b059d7149d84a5119ffe14aba
May 29 02:23:48.664028 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:48.663997    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerStarted","Data":"15b867eee5a3475c5bfc27ee63c25cb32bbcab310ffca0b01e158f60e6e6b486"}
May 29 02:23:48.664028 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:48.664030    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerStarted","Data":"66ac87b50e191b27ef4f82dcb0c96adadef5c14b059d7149d84a5119ffe14aba"}
May 29 02:23:52.676369 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:52.676334    2580 generic.go:358] "Generic (PLEG): container finished" podID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerID="15b867eee5a3475c5bfc27ee63c25cb32bbcab310ffca0b01e158f60e6e6b486" exitCode=0
May 29 02:23:52.676826 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:52.676393    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerDied","Data":"15b867eee5a3475c5bfc27ee63c25cb32bbcab310ffca0b01e158f60e6e6b486"}
May 29 02:23:53.680996 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:53.680957    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerStarted","Data":"4ee669da3482b057753fc7dd68b29fac1d9d9ec867f1a407a25629d29cd9b45a"}
May 29 02:23:53.862246 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:53.862196    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused"
May 29 02:23:54.485504 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:54.485454    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused"
May 29 02:23:58.696489 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:58.696455    2580 generic.go:358] "Generic (PLEG): container finished" podID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerID="4ee669da3482b057753fc7dd68b29fac1d9d9ec867f1a407a25629d29cd9b45a" exitCode=0
May 29 02:23:58.696922 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:58.696523    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerDied","Data":"4ee669da3482b057753fc7dd68b29fac1d9d9ec867f1a407a25629d29cd9b45a"}
May 29 02:23:59.703931 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:59.703886    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerStarted","Data":"def09495b461ea164dc354258ac61fc2549ad92571d5c104b13a2181e286c6b5"}
May 29 02:23:59.704574 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:59.703940    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerStarted","Data":"1b74f17ee0a03b9cd40d8acfaf42a4094c7d1108dccd0971ac8c1d55bbc39d0e"}
May 29 02:23:59.704574 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:59.703957    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerStarted","Data":"3439ae392566cf92dd19df7cd8eb2752e43618eca36e4b9418e4e01fdc4646ca"}
May 29 02:23:59.704574 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:59.703970    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerStarted","Data":"3f6ddab695bf9653fd273e72878289ce48cd77ee92d322c5f78fa05d093a1573"}
May 29 02:23:59.725830 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:23:59.725773    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podStartSLOduration=13.725753431 podStartE2EDuration="13.725753431s" podCreationTimestamp="2026-05-29 02:23:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:23:59.725635177 +0000 UTC m=+1536.308626083" watchObservedRunningTime="2026-05-29 02:23:59.725753431 +0000 UTC m=+1536.308744336"
May 29 02:24:02.601682 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:02.601641    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:02.602234 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:02.602209    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:02.602234 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:02.602238    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:02.602438 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:02.602252    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:02.603999 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:02.603962    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused"
May 29 02:24:02.604109 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:02.603996    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.39:8816: connect: connection refused"
May 29 02:24:02.604109 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:02.604031    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused"
May 29 02:24:02.604258 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:02.604112    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.39:6571: connect: connection refused"
May 29 02:24:03.862321 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:03.862275    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused"
May 29 02:24:04.485696 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:04.485657    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused"
May 29 02:24:05.602940 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.602908    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.603457 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.603174    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.603457 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.603312    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.603457 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.603340    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.603755 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.603737    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.603817 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.603793    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.603909 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.603891    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.607415 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.607398    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.725904 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.725862    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.725904 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.725906    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.726422 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.726406    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:05.728760 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:05.728741    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:24:13.863247 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:13.863200    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused"
May 29 02:24:13.863720 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:13.863473    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:24:14.485536 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:14.485495    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused"
May 29 02:24:14.485726 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:14.485654    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:24:16.762234 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.762147    2580 generic.go:358] "Generic (PLEG): container finished" podID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerID="070e553f644f21da0fbb0bdcd851d0741cb83345712804166138f3d77d9ccb4e" exitCode=137
May 29 02:24:16.762234 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.762171    2580 generic.go:358] "Generic (PLEG): container finished" podID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerID="e0236869988bb76c38a677d2b6c0cb52d29f03d8bdf5628b0eef63d618728563" exitCode=137
May 29 02:24:16.762234 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.762217    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerDied","Data":"070e553f644f21da0fbb0bdcd851d0741cb83345712804166138f3d77d9ccb4e"}
May 29 02:24:16.762673 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.762248    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerDied","Data":"e0236869988bb76c38a677d2b6c0cb52d29f03d8bdf5628b0eef63d618728563"}
May 29 02:24:16.943086 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.943058    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:24:16.971395 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.971370    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zw4t\" (UniqueName: \"kubernetes.io/projected/ecdba8be-61e8-4188-9657-4a5fca2e722b-kube-api-access-7zw4t\") pod \"ecdba8be-61e8-4188-9657-4a5fca2e722b\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") "
May 29 02:24:16.971509 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.971406    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-offline-tls\") pod \"ecdba8be-61e8-4188-9657-4a5fca2e722b\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") "
May 29 02:24:16.971509 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.971451    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-ui-tls\") pod \"ecdba8be-61e8-4188-9657-4a5fca2e722b\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") "
May 29 02:24:16.971509 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.971488    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-online-tls\") pod \"ecdba8be-61e8-4188-9657-4a5fca2e722b\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") "
May 29 02:24:16.971713 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.971529    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdba8be-61e8-4188-9657-4a5fca2e722b-feast-data\") pod \"ecdba8be-61e8-4188-9657-4a5fca2e722b\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") "
May 29 02:24:16.971713 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.971677    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-registry-tls\") pod \"ecdba8be-61e8-4188-9657-4a5fca2e722b\" (UID: \"ecdba8be-61e8-4188-9657-4a5fca2e722b\") "
May 29 02:24:16.972115 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.972077    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdba8be-61e8-4188-9657-4a5fca2e722b-feast-data" (OuterVolumeSpecName: "feast-data") pod "ecdba8be-61e8-4188-9657-4a5fca2e722b" (UID: "ecdba8be-61e8-4188-9657-4a5fca2e722b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:24:16.974335 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.974284    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-online-tls" (OuterVolumeSpecName: "online-tls") pod "ecdba8be-61e8-4188-9657-4a5fca2e722b" (UID: "ecdba8be-61e8-4188-9657-4a5fca2e722b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:24:16.974411 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.974351    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ecdba8be-61e8-4188-9657-4a5fca2e722b" (UID: "ecdba8be-61e8-4188-9657-4a5fca2e722b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:24:16.974411 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.974360    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ecdba8be-61e8-4188-9657-4a5fca2e722b" (UID: "ecdba8be-61e8-4188-9657-4a5fca2e722b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:24:16.974411 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.974381    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdba8be-61e8-4188-9657-4a5fca2e722b-kube-api-access-7zw4t" (OuterVolumeSpecName: "kube-api-access-7zw4t") pod "ecdba8be-61e8-4188-9657-4a5fca2e722b" (UID: "ecdba8be-61e8-4188-9657-4a5fca2e722b"). InnerVolumeSpecName "kube-api-access-7zw4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:24:16.974524 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:16.974429    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "ecdba8be-61e8-4188-9657-4a5fca2e722b" (UID: "ecdba8be-61e8-4188-9657-4a5fca2e722b"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:24:17.072782 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.072740    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:24:17.072782 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.072783    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdba8be-61e8-4188-9657-4a5fca2e722b-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:24:17.072782 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.072794    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:24:17.073010 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.072803    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zw4t\" (UniqueName: \"kubernetes.io/projected/ecdba8be-61e8-4188-9657-4a5fca2e722b-kube-api-access-7zw4t\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:24:17.073010 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.072813    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:24:17.073010 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.072823    2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ecdba8be-61e8-4188-9657-4a5fca2e722b-ui-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:24:17.767209 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.767170    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh" event={"ID":"ecdba8be-61e8-4188-9657-4a5fca2e722b","Type":"ContainerDied","Data":"a5a468151ceef40ad68eec22f433b23b1348cddb448a309da0a2c0d787d1e55c"}
May 29 02:24:17.767209 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.767204    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"
May 29 02:24:17.767760 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.767222    2580 scope.go:117] "RemoveContainer" containerID="869b13ddb431b051bcaac6302501ff5ccc2412d28c8397683f7a978f02716d73"
May 29 02:24:17.776097 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.776073    2580 scope.go:117] "RemoveContainer" containerID="070e553f644f21da0fbb0bdcd851d0741cb83345712804166138f3d77d9ccb4e"
May 29 02:24:17.783259 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.783240    2580 scope.go:117] "RemoveContainer" containerID="06820b2458144f304b7d029f6bc83e0bf2066b50d93d76ef3e4b459d355bf342"
May 29 02:24:17.789275 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.789254    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"]
May 29 02:24:17.790176 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.790161    2580 scope.go:117] "RemoveContainer" containerID="e0236869988bb76c38a677d2b6c0cb52d29f03d8bdf5628b0eef63d618728563"
May 29 02:24:17.794458 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.794429    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n8mwh"]
May 29 02:24:17.797360 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.797343    2580 scope.go:117] "RemoveContainer" containerID="f79b1073dfb60c8e22ab20e5b3f83b60b9f62b0120a44a2e55c45ad02d9b821d"
May 29 02:24:17.804379 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.804363    2580 scope.go:117] "RemoveContainer" containerID="adac5c3ecf43c35905a95c025a401c6d2d570c9c13b3bfd9df00e1da21f44c82"
May 29 02:24:17.978242 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:24:17.978210    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" path="/var/lib/kubelet/pods/ecdba8be-61e8-4188-9657-4a5fca2e722b/volumes"
May 29 02:25:08.019941 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.019898    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"]
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020288    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="offline"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020305    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="offline"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020323    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="feast-init"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020332    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="feast-init"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020344    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="feast-apply"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020351    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="feast-apply"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020373    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="ui"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020378    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="ui"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020387    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="registry"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020392    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="registry"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020399    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="online"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020403    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="online"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020455    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="ui"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020466    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="registry"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020473    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="offline"
May 29 02:25:08.022641 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.020481    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ecdba8be-61e8-4188-9657-4a5fca2e722b" containerName="online"
May 29 02:25:08.023441 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.023422    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.025459 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.025439    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 29 02:25:08.025574 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.025455    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 29 02:25:08.025574 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.025458    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 29 02:25:08.025725 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.025710    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 29 02:25:08.025820 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.025800    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-ggfx7\""
May 29 02:25:08.025877 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.025819    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 29 02:25:08.025917 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.025857    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 29 02:25:08.034326 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.034306    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"]
May 29 02:25:08.069765 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.069743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.069882 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.069787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.069882 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.069808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13c5e759-6178-432b-b9af-8acb774bba92-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.069958 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.069898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13c5e759-6178-432b-b9af-8acb774bba92-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.069958 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.069937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.070025 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.069956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44mc2\" (UniqueName: \"kubernetes.io/projected/13c5e759-6178-432b-b9af-8acb774bba92-kube-api-access-44mc2\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.170508 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.170476    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.170682 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.170515    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13c5e759-6178-432b-b9af-8acb774bba92-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.170682 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.170572    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13c5e759-6178-432b-b9af-8acb774bba92-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.170682 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.170629    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.170682 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:25:08.170644    2580 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 02:25:08.170682 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.170654    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44mc2\" (UniqueName: \"kubernetes.io/projected/13c5e759-6178-432b-b9af-8acb774bba92-kube-api-access-44mc2\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.170682 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.170679    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.170988 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:25:08.170720    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-online-tls podName:13c5e759-6178-432b-b9af-8acb774bba92 nodeName:}" failed. No retries permitted until 2026-05-29 02:25:08.670698188 +0000 UTC m=+1605.253689085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-online-tls") pod "feast-simple-feast-remote-setup-5cff94746b-r6wrh" (UID: "13c5e759-6178-432b-b9af-8acb774bba92") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 29 02:25:08.171119 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.171098    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13c5e759-6178-432b-b9af-8acb774bba92-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.171260 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.171240    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13c5e759-6178-432b-b9af-8acb774bba92-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.173013 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.172993    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.173116 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.173079    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.180585 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.180564    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44mc2\" (UniqueName: \"kubernetes.io/projected/13c5e759-6178-432b-b9af-8acb774bba92-kube-api-access-44mc2\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.673336 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.673273    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.675621 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.675583    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-r6wrh\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:08.932132 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:08.932040    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:09.056808 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:09.056784    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"]
May 29 02:25:09.058672 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:25:09.058646    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13c5e759_6178_432b_b9af_8acb774bba92.slice/crio-fa1c853b99a09210d2059c5bb565f4e9caad3da2a0b79eb718d882e85a0387f1 WatchSource:0}: Error finding container fa1c853b99a09210d2059c5bb565f4e9caad3da2a0b79eb718d882e85a0387f1: Status 404 returned error can't find the container with id fa1c853b99a09210d2059c5bb565f4e9caad3da2a0b79eb718d882e85a0387f1
May 29 02:25:09.912862 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:09.912830    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerStarted","Data":"0ef0c358484f3d2e10327668f37e6b7e9fbb204657a7a14c8fff7ab384409a95"}
May 29 02:25:09.912862 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:09.912865    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerStarted","Data":"fa1c853b99a09210d2059c5bb565f4e9caad3da2a0b79eb718d882e85a0387f1"}
May 29 02:25:13.925342 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:13.925253    2580 generic.go:358] "Generic (PLEG): container finished" podID="13c5e759-6178-432b-b9af-8acb774bba92" containerID="0ef0c358484f3d2e10327668f37e6b7e9fbb204657a7a14c8fff7ab384409a95" exitCode=0
May 29 02:25:13.925342 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:13.925303    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerDied","Data":"0ef0c358484f3d2e10327668f37e6b7e9fbb204657a7a14c8fff7ab384409a95"}
May 29 02:25:14.930909 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:14.930866    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerStarted","Data":"a21e46d7293258d78a0d83ba09b8343c4f90f4856ad4269ec5005bff1044fb99"}
May 29 02:25:19.946333 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:19.946299    2580 generic.go:358] "Generic (PLEG): container finished" podID="13c5e759-6178-432b-b9af-8acb774bba92" containerID="a21e46d7293258d78a0d83ba09b8343c4f90f4856ad4269ec5005bff1044fb99" exitCode=0
May 29 02:25:19.946780 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:19.946353    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerDied","Data":"a21e46d7293258d78a0d83ba09b8343c4f90f4856ad4269ec5005bff1044fb99"}
May 29 02:25:20.953007 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:20.952970    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerStarted","Data":"ce1fc050af9d8a07c600ca7bd340b862dcc4ccb29d4b6183d99887d664f8ee58"}
May 29 02:25:20.953007 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:20.953014    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerStarted","Data":"9936e4d9ac87982e2bd27ff853d48daab5381e5447568e571ee618902ff96748"}
May 29 02:25:20.953583 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:20.953025    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerStarted","Data":"8f5c9f6f54112718d54ca36369018d6248c2ee5d0045a953000f267706c96005"}
May 29 02:25:20.978516 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:20.978458    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podStartSLOduration=12.978437584 podStartE2EDuration="12.978437584s" podCreationTimestamp="2026-05-29 02:25:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:25:20.976585563 +0000 UTC m=+1617.559576471" watchObservedRunningTime="2026-05-29 02:25:20.978437584 +0000 UTC m=+1617.561428490"
May 29 02:25:23.932613 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:23.932553    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:23.933284 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:23.933258    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:23.933284 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:23.933288    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:23.934262 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:23.934228    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused"
May 29 02:25:23.934374 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:23.934265    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.40:8816: connect: connection refused"
May 29 02:25:23.934374 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:23.934281    2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused"
May 29 02:25:26.933471 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:26.933442    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:26.933946 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:26.933571    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:26.934022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:26.933973    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:26.934022 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:26.934003    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:26.934172 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:26.934155    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:26.934369 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:26.934356    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:26.937300 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:26.937281    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:26.976847 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:26.976817    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:25:26.979703 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:25:26.979670    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:26:28.989328 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:28.989288    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"]
May 29 02:26:28.989966 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:28.989749    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="online" containerID="cri-o://8f5c9f6f54112718d54ca36369018d6248c2ee5d0045a953000f267706c96005" gracePeriod=30
May 29 02:26:28.990221 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:28.990167    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="ui" containerID="cri-o://ce1fc050af9d8a07c600ca7bd340b862dcc4ccb29d4b6183d99887d664f8ee58" gracePeriod=30
May 29 02:26:28.990319 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:28.990288    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="offline" containerID="cri-o://9936e4d9ac87982e2bd27ff853d48daab5381e5447568e571ee618902ff96748" gracePeriod=30
May 29 02:26:29.125191 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:29.125154    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"]
May 29 02:26:29.125796 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:29.125735    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="registry" containerID="cri-o://3f6ddab695bf9653fd273e72878289ce48cd77ee92d322c5f78fa05d093a1573" gracePeriod=30
May 29 02:26:29.126352 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:29.125752    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="ui" containerID="cri-o://def09495b461ea164dc354258ac61fc2549ad92571d5c104b13a2181e286c6b5" gracePeriod=30
May 29 02:26:29.126486 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:29.125793    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="offline" containerID="cri-o://1b74f17ee0a03b9cd40d8acfaf42a4094c7d1108dccd0971ac8c1d55bbc39d0e" gracePeriod=30
May 29 02:26:29.126486 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:29.125817    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="online" containerID="cri-o://3439ae392566cf92dd19df7cd8eb2752e43618eca36e4b9418e4e01fdc4646ca" gracePeriod=30
May 29 02:26:30.158571 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:30.158481    2580 generic.go:358] "Generic (PLEG): container finished" podID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerID="def09495b461ea164dc354258ac61fc2549ad92571d5c104b13a2181e286c6b5" exitCode=0
May 29 02:26:30.158571 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:30.158510    2580 generic.go:358] "Generic (PLEG): container finished" podID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerID="3439ae392566cf92dd19df7cd8eb2752e43618eca36e4b9418e4e01fdc4646ca" exitCode=0
May 29 02:26:30.158571 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:30.158556    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerDied","Data":"def09495b461ea164dc354258ac61fc2549ad92571d5c104b13a2181e286c6b5"}
May 29 02:26:30.159048 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:30.158610    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerDied","Data":"3439ae392566cf92dd19df7cd8eb2752e43618eca36e4b9418e4e01fdc4646ca"}
May 29 02:26:30.160870 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:30.160839    2580 generic.go:358] "Generic (PLEG): container finished" podID="13c5e759-6178-432b-b9af-8acb774bba92" containerID="ce1fc050af9d8a07c600ca7bd340b862dcc4ccb29d4b6183d99887d664f8ee58" exitCode=0
May 29 02:26:30.160870 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:30.160863    2580 generic.go:358] "Generic (PLEG): container finished" podID="13c5e759-6178-432b-b9af-8acb774bba92" containerID="8f5c9f6f54112718d54ca36369018d6248c2ee5d0045a953000f267706c96005" exitCode=0
May 29 02:26:30.161040 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:30.160879    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerDied","Data":"ce1fc050af9d8a07c600ca7bd340b862dcc4ccb29d4b6183d99887d664f8ee58"}
May 29 02:26:30.161040 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:30.160911    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerDied","Data":"8f5c9f6f54112718d54ca36369018d6248c2ee5d0045a953000f267706c96005"}
May 29 02:26:35.726359 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:35.726316    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused"
May 29 02:26:35.726837 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:35.726361    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused"
May 29 02:26:36.934365 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:36.934325    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused"
May 29 02:26:36.977203 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:36.977167    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused"
May 29 02:26:45.726166 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:45.726080    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused"
May 29 02:26:45.726166 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:45.726130    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused"
May 29 02:26:46.934243 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:46.934192    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused"
May 29 02:26:46.977944 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:46.977913    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused"
May 29 02:26:55.726483 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:55.726428    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused"
May 29 02:26:55.726483 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:55.726453    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused"
May 29 02:26:55.726935 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:55.726622    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:26:55.726935 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:55.726683    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:26:56.934128 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:56.934083    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused"
May 29 02:26:56.934537 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:56.934221    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:26:56.977100 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:56.977065    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused"
May 29 02:26:56.977253 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:56.977183    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:26:59.245910 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.245829    2580 generic.go:358] "Generic (PLEG): container finished" podID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerID="1b74f17ee0a03b9cd40d8acfaf42a4094c7d1108dccd0971ac8c1d55bbc39d0e" exitCode=137
May 29 02:26:59.245910 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.245853    2580 generic.go:358] "Generic (PLEG): container finished" podID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerID="3f6ddab695bf9653fd273e72878289ce48cd77ee92d322c5f78fa05d093a1573" exitCode=137
May 29 02:26:59.245910 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.245900    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerDied","Data":"1b74f17ee0a03b9cd40d8acfaf42a4094c7d1108dccd0971ac8c1d55bbc39d0e"}
May 29 02:26:59.246337 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.245938    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerDied","Data":"3f6ddab695bf9653fd273e72878289ce48cd77ee92d322c5f78fa05d093a1573"}
May 29 02:26:59.248035 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.248015    2580 generic.go:358] "Generic (PLEG): container finished" podID="13c5e759-6178-432b-b9af-8acb774bba92" containerID="9936e4d9ac87982e2bd27ff853d48daab5381e5447568e571ee618902ff96748" exitCode=137
May 29 02:26:59.248136 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.248048    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerDied","Data":"9936e4d9ac87982e2bd27ff853d48daab5381e5447568e571ee618902ff96748"}
May 29 02:26:59.631905 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.631881    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:26:59.724092 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.724062    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-online-tls\") pod \"13c5e759-6178-432b-b9af-8acb774bba92\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") "
May 29 02:26:59.724244 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.724102    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-ui-tls\") pod \"13c5e759-6178-432b-b9af-8acb774bba92\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") "
May 29 02:26:59.724244 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.724183    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13c5e759-6178-432b-b9af-8acb774bba92-feast-data\") pod \"13c5e759-6178-432b-b9af-8acb774bba92\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") "
May 29 02:26:59.724244 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.724226    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-offline-tls\") pod \"13c5e759-6178-432b-b9af-8acb774bba92\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") "
May 29 02:26:59.724414 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.724279    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13c5e759-6178-432b-b9af-8acb774bba92-registry-tls\") pod \"13c5e759-6178-432b-b9af-8acb774bba92\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") "
May 29 02:26:59.724414 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.724315    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44mc2\" (UniqueName: \"kubernetes.io/projected/13c5e759-6178-432b-b9af-8acb774bba92-kube-api-access-44mc2\") pod \"13c5e759-6178-432b-b9af-8acb774bba92\" (UID: \"13c5e759-6178-432b-b9af-8acb774bba92\") "
May 29 02:26:59.724701 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.724671    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c5e759-6178-432b-b9af-8acb774bba92-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "13c5e759-6178-432b-b9af-8acb774bba92" (UID: "13c5e759-6178-432b-b9af-8acb774bba92"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 02:26:59.724784 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.724721    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c5e759-6178-432b-b9af-8acb774bba92-feast-data" (OuterVolumeSpecName: "feast-data") pod "13c5e759-6178-432b-b9af-8acb774bba92" (UID: "13c5e759-6178-432b-b9af-8acb774bba92"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:26:59.726427 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.726401    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-online-tls" (OuterVolumeSpecName: "online-tls") pod "13c5e759-6178-432b-b9af-8acb774bba92" (UID: "13c5e759-6178-432b-b9af-8acb774bba92"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:26:59.726638 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.726614    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "13c5e759-6178-432b-b9af-8acb774bba92" (UID: "13c5e759-6178-432b-b9af-8acb774bba92"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:26:59.726638 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.726628    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "13c5e759-6178-432b-b9af-8acb774bba92" (UID: "13c5e759-6178-432b-b9af-8acb774bba92"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:26:59.726750 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.726632    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c5e759-6178-432b-b9af-8acb774bba92-kube-api-access-44mc2" (OuterVolumeSpecName: "kube-api-access-44mc2") pod "13c5e759-6178-432b-b9af-8acb774bba92" (UID: "13c5e759-6178-432b-b9af-8acb774bba92"). InnerVolumeSpecName "kube-api-access-44mc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:26:59.764947 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.764928    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:26:59.824975 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.824948    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-registry-tls\") pod \"4e252561-bcdc-4edc-88e5-d40b05720cb1\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") "
May 29 02:26:59.825139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.824983    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-ui-tls\") pod \"4e252561-bcdc-4edc-88e5-d40b05720cb1\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") "
May 29 02:26:59.825139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825012    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-offline-tls\") pod \"4e252561-bcdc-4edc-88e5-d40b05720cb1\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") "
May 29 02:26:59.825139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825062    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4e252561-bcdc-4edc-88e5-d40b05720cb1-feast-data\") pod \"4e252561-bcdc-4edc-88e5-d40b05720cb1\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") "
May 29 02:26:59.825139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825086    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-online-tls\") pod \"4e252561-bcdc-4edc-88e5-d40b05720cb1\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") "
May 29 02:26:59.825139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825120    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnnlz\" (UniqueName: \"kubernetes.io/projected/4e252561-bcdc-4edc-88e5-d40b05720cb1-kube-api-access-nnnlz\") pod \"4e252561-bcdc-4edc-88e5-d40b05720cb1\" (UID: \"4e252561-bcdc-4edc-88e5-d40b05720cb1\") "
May 29 02:26:59.825403 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825333    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44mc2\" (UniqueName: \"kubernetes.io/projected/13c5e759-6178-432b-b9af-8acb774bba92-kube-api-access-44mc2\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.825403 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825350    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.825403 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825366    2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-ui-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.825403 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825379    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13c5e759-6178-432b-b9af-8acb774bba92-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.825403 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825392    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13c5e759-6178-432b-b9af-8acb774bba92-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.825682 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825405    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13c5e759-6178-432b-b9af-8acb774bba92-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.825682 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.825654    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e252561-bcdc-4edc-88e5-d40b05720cb1-feast-data" (OuterVolumeSpecName: "feast-data") pod "4e252561-bcdc-4edc-88e5-d40b05720cb1" (UID: "4e252561-bcdc-4edc-88e5-d40b05720cb1"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:26:59.827145 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.827116    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4e252561-bcdc-4edc-88e5-d40b05720cb1" (UID: "4e252561-bcdc-4edc-88e5-d40b05720cb1"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:26:59.827145 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.827138    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4e252561-bcdc-4edc-88e5-d40b05720cb1" (UID: "4e252561-bcdc-4edc-88e5-d40b05720cb1"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:26:59.827283 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.827145    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e252561-bcdc-4edc-88e5-d40b05720cb1-kube-api-access-nnnlz" (OuterVolumeSpecName: "kube-api-access-nnnlz") pod "4e252561-bcdc-4edc-88e5-d40b05720cb1" (UID: "4e252561-bcdc-4edc-88e5-d40b05720cb1"). InnerVolumeSpecName "kube-api-access-nnnlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:26:59.827487 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.827465    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4e252561-bcdc-4edc-88e5-d40b05720cb1" (UID: "4e252561-bcdc-4edc-88e5-d40b05720cb1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:26:59.827487 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.827478    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-online-tls" (OuterVolumeSpecName: "online-tls") pod "4e252561-bcdc-4edc-88e5-d40b05720cb1" (UID: "4e252561-bcdc-4edc-88e5-d40b05720cb1"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 02:26:59.925887 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.925850    2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4e252561-bcdc-4edc-88e5-d40b05720cb1-feast-data\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.925887 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.925882    2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-online-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.926067 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.925896    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnnlz\" (UniqueName: \"kubernetes.io/projected/4e252561-bcdc-4edc-88e5-d40b05720cb1-kube-api-access-nnnlz\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.926067 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.925909    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-registry-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.926067 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.925920    2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-ui-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:26:59.926067 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:26:59.925931    2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4e252561-bcdc-4edc-88e5-d40b05720cb1-offline-tls\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:27:00.258214 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.258178    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9" event={"ID":"4e252561-bcdc-4edc-88e5-d40b05720cb1","Type":"ContainerDied","Data":"66ac87b50e191b27ef4f82dcb0c96adadef5c14b059d7149d84a5119ffe14aba"}
May 29 02:27:00.258214 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.258211    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"
May 29 02:27:00.258737 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.258231    2580 scope.go:117] "RemoveContainer" containerID="def09495b461ea164dc354258ac61fc2549ad92571d5c104b13a2181e286c6b5"
May 29 02:27:00.260907 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.260876    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh" event={"ID":"13c5e759-6178-432b-b9af-8acb774bba92","Type":"ContainerDied","Data":"fa1c853b99a09210d2059c5bb565f4e9caad3da2a0b79eb718d882e85a0387f1"}
May 29 02:27:00.261036 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.260889    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"
May 29 02:27:00.267229 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.267209    2580 scope.go:117] "RemoveContainer" containerID="1b74f17ee0a03b9cd40d8acfaf42a4094c7d1108dccd0971ac8c1d55bbc39d0e"
May 29 02:27:00.274171 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.274150    2580 scope.go:117] "RemoveContainer" containerID="3439ae392566cf92dd19df7cd8eb2752e43618eca36e4b9418e4e01fdc4646ca"
May 29 02:27:00.280679 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.280655    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"]
May 29 02:27:00.282216 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.282197    2580 scope.go:117] "RemoveContainer" containerID="3f6ddab695bf9653fd273e72878289ce48cd77ee92d322c5f78fa05d093a1573"
May 29 02:27:00.284867 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.284847    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-54wz9"]
May 29 02:27:00.289172 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.289156    2580 scope.go:117] "RemoveContainer" containerID="4ee669da3482b057753fc7dd68b29fac1d9d9ec867f1a407a25629d29cd9b45a"
May 29 02:27:00.296093 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.296071    2580 scope.go:117] "RemoveContainer" containerID="15b867eee5a3475c5bfc27ee63c25cb32bbcab310ffca0b01e158f60e6e6b486"
May 29 02:27:00.297252 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.297230    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"]
May 29 02:27:00.302455 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.302436    2580 scope.go:117] "RemoveContainer" containerID="ce1fc050af9d8a07c600ca7bd340b862dcc4ccb29d4b6183d99887d664f8ee58"
May 29 02:27:00.310101 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.310085    2580 scope.go:117] "RemoveContainer" containerID="9936e4d9ac87982e2bd27ff853d48daab5381e5447568e571ee618902ff96748"
May 29 02:27:00.311201 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.311181    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-r6wrh"]
May 29 02:27:00.316461 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.316445    2580 scope.go:117] "RemoveContainer" containerID="8f5c9f6f54112718d54ca36369018d6248c2ee5d0045a953000f267706c96005"
May 29 02:27:00.322773 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.322758    2580 scope.go:117] "RemoveContainer" containerID="a21e46d7293258d78a0d83ba09b8343c4f90f4856ad4269ec5005bff1044fb99"
May 29 02:27:00.329161 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:00.329145    2580 scope.go:117] "RemoveContainer" containerID="0ef0c358484f3d2e10327668f37e6b7e9fbb204657a7a14c8fff7ab384409a95"
May 29 02:27:01.975069 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:01.975036    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c5e759-6178-432b-b9af-8acb774bba92" path="/var/lib/kubelet/pods/13c5e759-6178-432b-b9af-8acb774bba92/volumes"
May 29 02:27:01.975570 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:01.975558    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" path="/var/lib/kubelet/pods/4e252561-bcdc-4edc-88e5-d40b05720cb1/volumes"
May 29 02:27:18.145081 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145033    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v45s5/must-gather-ch6p7"]
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145410    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="online"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145427    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="online"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145444    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="online"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145450    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="online"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145459    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="feast-init"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145468    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="feast-init"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145474    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="offline"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145479    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="offline"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145489    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="offline"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145494    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="offline"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145500    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="ui"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145505    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="ui"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145510    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="feast-apply"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145515    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="feast-apply"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145523    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="ui"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145527    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="ui"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145533    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="registry"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145538    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="registry"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145544    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="feast-init"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145549    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="feast-init"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145554    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="feast-apply"
May 29 02:27:18.145584 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145559    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="feast-apply"
May 29 02:27:18.146272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145619    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="ui"
May 29 02:27:18.146272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145626    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="ui"
May 29 02:27:18.146272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145633    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="registry"
May 29 02:27:18.146272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145639    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="online"
May 29 02:27:18.146272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145645    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="online"
May 29 02:27:18.146272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145651    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="13c5e759-6178-432b-b9af-8acb774bba92" containerName="offline"
May 29 02:27:18.146272 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.145657    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e252561-bcdc-4edc-88e5-d40b05720cb1" containerName="offline"
May 29 02:27:18.150487 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.150467    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:18.152846 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.152825    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v45s5\"/\"openshift-service-ca.crt\""
May 29 02:27:18.153485 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.153467    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v45s5\"/\"default-dockercfg-2b9pv\""
May 29 02:27:18.153565 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.153476    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v45s5\"/\"kube-root-ca.crt\""
May 29 02:27:18.159375 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.159351    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v45s5/must-gather-ch6p7"]
May 29 02:27:18.274767 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.274733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clvpj\" (UniqueName: \"kubernetes.io/projected/e5a11bbd-4699-4010-be1e-9ff591918114-kube-api-access-clvpj\") pod \"must-gather-ch6p7\" (UID: \"e5a11bbd-4699-4010-be1e-9ff591918114\") " pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:18.274940 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.274806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5a11bbd-4699-4010-be1e-9ff591918114-must-gather-output\") pod \"must-gather-ch6p7\" (UID: \"e5a11bbd-4699-4010-be1e-9ff591918114\") " pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:18.375908 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.375853    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5a11bbd-4699-4010-be1e-9ff591918114-must-gather-output\") pod \"must-gather-ch6p7\" (UID: \"e5a11bbd-4699-4010-be1e-9ff591918114\") " pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:18.376086 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.375939    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-clvpj\" (UniqueName: \"kubernetes.io/projected/e5a11bbd-4699-4010-be1e-9ff591918114-kube-api-access-clvpj\") pod \"must-gather-ch6p7\" (UID: \"e5a11bbd-4699-4010-be1e-9ff591918114\") " pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:18.376210 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.376190    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5a11bbd-4699-4010-be1e-9ff591918114-must-gather-output\") pod \"must-gather-ch6p7\" (UID: \"e5a11bbd-4699-4010-be1e-9ff591918114\") " pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:18.388467 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.388435    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvpj\" (UniqueName: \"kubernetes.io/projected/e5a11bbd-4699-4010-be1e-9ff591918114-kube-api-access-clvpj\") pod \"must-gather-ch6p7\" (UID: \"e5a11bbd-4699-4010-be1e-9ff591918114\") " pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:18.471614 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.471483    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:18.595701 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:18.595668    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v45s5/must-gather-ch6p7"]
May 29 02:27:18.599244 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:27:18.599214    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5a11bbd_4699_4010_be1e_9ff591918114.slice/crio-dc91f64ef7ea56b1e3040c741133ec0a6277c9ae8d405a4987ee54575d83d705 WatchSource:0}: Error finding container dc91f64ef7ea56b1e3040c741133ec0a6277c9ae8d405a4987ee54575d83d705: Status 404 returned error can't find the container with id dc91f64ef7ea56b1e3040c741133ec0a6277c9ae8d405a4987ee54575d83d705
May 29 02:27:19.320959 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:19.320923    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v45s5/must-gather-ch6p7" event={"ID":"e5a11bbd-4699-4010-be1e-9ff591918114","Type":"ContainerStarted","Data":"dc91f64ef7ea56b1e3040c741133ec0a6277c9ae8d405a4987ee54575d83d705"}
May 29 02:27:25.341139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:25.341101    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v45s5/must-gather-ch6p7" event={"ID":"e5a11bbd-4699-4010-be1e-9ff591918114","Type":"ContainerStarted","Data":"ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab"}
May 29 02:27:25.341139 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:25.341139    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v45s5/must-gather-ch6p7" event={"ID":"e5a11bbd-4699-4010-be1e-9ff591918114","Type":"ContainerStarted","Data":"17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62"}
May 29 02:27:25.357996 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:25.357940    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-v45s5/must-gather-ch6p7" podStartSLOduration=1.181443164 podStartE2EDuration="7.357925343s" podCreationTimestamp="2026-05-29 02:27:18 +0000 UTC" firstStartedPulling="2026-05-29 02:27:18.600791016 +0000 UTC m=+1735.183781898" lastFinishedPulling="2026-05-29 02:27:24.777273192 +0000 UTC m=+1741.360264077" observedRunningTime="2026-05-29 02:27:25.357077967 +0000 UTC m=+1741.940068873" watchObservedRunningTime="2026-05-29 02:27:25.357925343 +0000 UTC m=+1741.940916247"
May 29 02:27:34.373747 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:34.373711    2580 generic.go:358] "Generic (PLEG): container finished" podID="e5a11bbd-4699-4010-be1e-9ff591918114" containerID="17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62" exitCode=0
May 29 02:27:34.374159 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:34.373786    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v45s5/must-gather-ch6p7" event={"ID":"e5a11bbd-4699-4010-be1e-9ff591918114","Type":"ContainerDied","Data":"17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62"}
May 29 02:27:34.374159 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:34.374098    2580 scope.go:117] "RemoveContainer" containerID="17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62"
May 29 02:27:35.334269 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:35.334239    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-v45s5_must-gather-ch6p7_e5a11bbd-4699-4010-be1e-9ff591918114/gather/0.log"
May 29 02:27:38.413064 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:38.413036    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vb52g_c6c5c410-be0e-4f48-9ace-04f0312622a5/global-pull-secret-syncer/0.log"
May 29 02:27:38.531419 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:38.531387    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zsdgm_58f43f17-039c-4be9-9db1-d1fb13349a95/konnectivity-agent/0.log"
May 29 02:27:38.549412 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:38.549388    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-62.ec2.internal_6bfc788d18bccc31168708c81f074e78/haproxy/0.log"
May 29 02:27:40.681925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.681892    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-v45s5/must-gather-ch6p7"]
May 29 02:27:40.682332 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.682093    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-v45s5/must-gather-ch6p7" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" containerName="copy" containerID="cri-o://ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab" gracePeriod=2
May 29 02:27:40.686519 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.686432    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-v45s5/must-gather-ch6p7"]
May 29 02:27:40.907485 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.907462    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-v45s5_must-gather-ch6p7_e5a11bbd-4699-4010-be1e-9ff591918114/copy/0.log"
May 29 02:27:40.907820 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.907807    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:40.912356 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.912331    2580 status_manager.go:895] "Failed to get status for pod" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" pod="openshift-must-gather-v45s5/must-gather-ch6p7" err="pods \"must-gather-ch6p7\" is forbidden: User \"system:node:ip-10-0-129-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-v45s5\": no relationship found between node 'ip-10-0-129-62.ec2.internal' and this object"
May 29 02:27:40.963668 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.963576    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5a11bbd-4699-4010-be1e-9ff591918114-must-gather-output\") pod \"e5a11bbd-4699-4010-be1e-9ff591918114\" (UID: \"e5a11bbd-4699-4010-be1e-9ff591918114\") "
May 29 02:27:40.963668 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.963662    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clvpj\" (UniqueName: \"kubernetes.io/projected/e5a11bbd-4699-4010-be1e-9ff591918114-kube-api-access-clvpj\") pod \"e5a11bbd-4699-4010-be1e-9ff591918114\" (UID: \"e5a11bbd-4699-4010-be1e-9ff591918114\") "
May 29 02:27:40.963925 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.963900    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a11bbd-4699-4010-be1e-9ff591918114-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e5a11bbd-4699-4010-be1e-9ff591918114" (UID: "e5a11bbd-4699-4010-be1e-9ff591918114"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 02:27:40.965802 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:40.965782    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a11bbd-4699-4010-be1e-9ff591918114-kube-api-access-clvpj" (OuterVolumeSpecName: "kube-api-access-clvpj") pod "e5a11bbd-4699-4010-be1e-9ff591918114" (UID: "e5a11bbd-4699-4010-be1e-9ff591918114"). InnerVolumeSpecName "kube-api-access-clvpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 02:27:41.065057 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.065019    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clvpj\" (UniqueName: \"kubernetes.io/projected/e5a11bbd-4699-4010-be1e-9ff591918114-kube-api-access-clvpj\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:27:41.065057 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.065047    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5a11bbd-4699-4010-be1e-9ff591918114-must-gather-output\") on node \"ip-10-0-129-62.ec2.internal\" DevicePath \"\""
May 29 02:27:41.394118 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.394091    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-v45s5_must-gather-ch6p7_e5a11bbd-4699-4010-be1e-9ff591918114/copy/0.log"
May 29 02:27:41.394426 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.394403    2580 generic.go:358] "Generic (PLEG): container finished" podID="e5a11bbd-4699-4010-be1e-9ff591918114" containerID="ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab" exitCode=143
May 29 02:27:41.394483 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.394450    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v45s5/must-gather-ch6p7"
May 29 02:27:41.394483 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.394465    2580 scope.go:117] "RemoveContainer" containerID="ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab"
May 29 02:27:41.397212 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.397179    2580 status_manager.go:895] "Failed to get status for pod" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" pod="openshift-must-gather-v45s5/must-gather-ch6p7" err="pods \"must-gather-ch6p7\" is forbidden: User \"system:node:ip-10-0-129-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-v45s5\": no relationship found between node 'ip-10-0-129-62.ec2.internal' and this object"
May 29 02:27:41.403977 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.403953    2580 scope.go:117] "RemoveContainer" containerID="17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62"
May 29 02:27:41.407579 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.407547    2580 status_manager.go:895] "Failed to get status for pod" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" pod="openshift-must-gather-v45s5/must-gather-ch6p7" err="pods \"must-gather-ch6p7\" is forbidden: User \"system:node:ip-10-0-129-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-v45s5\": no relationship found between node 'ip-10-0-129-62.ec2.internal' and this object"
May 29 02:27:41.414809 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.414792    2580 scope.go:117] "RemoveContainer" containerID="ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab"
May 29 02:27:41.415064 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:27:41.415047    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab\": container with ID starting with ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab not found: ID does not exist" containerID="ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab"
May 29 02:27:41.415114 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.415073    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab"} err="failed to get container status \"ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab\": rpc error: code = NotFound desc = could not find container \"ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab\": container with ID starting with ccc9ec87d183e984f6a679e0b95ba8d6a2c0e25382d5c12c209605ca67244dab not found: ID does not exist"
May 29 02:27:41.415114 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.415092    2580 scope.go:117] "RemoveContainer" containerID="17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62"
May 29 02:27:41.415310 ip-10-0-129-62 kubenswrapper[2580]: E0529 02:27:41.415294    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62\": container with ID starting with 17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62 not found: ID does not exist" containerID="17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62"
May 29 02:27:41.415355 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.415315    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62"} err="failed to get container status \"17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62\": rpc error: code = NotFound desc = could not find container \"17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62\": container with ID starting with 17ef490f268ccecfc1ee75f1451647b510549757b1c78366110ecac670878a62 not found: ID does not exist"
May 29 02:27:41.655928 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.655848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bb8bb4fc7-2n67j_ebb96298-ed02-4b73-9224-52a8e4eb19cb/metrics-server/0.log"
May 29 02:27:41.864257 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.864231    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s9kqs_da4ada43-7e96-4292-912b-d4c4d45d48c8/node-exporter/0.log"
May 29 02:27:41.883035 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.883010    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s9kqs_da4ada43-7e96-4292-912b-d4c4d45d48c8/kube-rbac-proxy/0.log"
May 29 02:27:41.900469 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.900445    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s9kqs_da4ada43-7e96-4292-912b-d4c4d45d48c8/init-textfile/0.log"
May 29 02:27:41.974417 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:41.974325    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" path="/var/lib/kubelet/pods/e5a11bbd-4699-4010-be1e-9ff591918114/volumes"
May 29 02:27:42.332552 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:42.332517    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-tdsdc_a198136c-fc63-42cb-9bb2-ae343cbad7c2/prometheus-operator/0.log"
May 29 02:27:42.359512 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:42.359487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-tdsdc_a198136c-fc63-42cb-9bb2-ae343cbad7c2/kube-rbac-proxy/0.log"
May 29 02:27:42.385673 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:42.385649    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-m25c4_94d3dcda-9c06-490b-9c32-ad1cab289bd1/prometheus-operator-admission-webhook/0.log"
May 29 02:27:44.123403 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.123371    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/1.log"
May 29 02:27:44.127722 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.127702    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2zz99_146b917f-a47c-45db-a65c-33c5ecaf1237/console-operator/2.log"
May 29 02:27:44.479549 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.479469    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-j4cnx_6f6e5a54-bc55-4252-9f1d-657a5fb7c340/download-server/0.log"
May 29 02:27:44.845880 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.845832    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-kbzqb_9b07acd8-0d6c-4c1d-8732-1464c1fba503/volume-data-source-validator/0.log"
May 29 02:27:44.922220 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.922186    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"]
May 29 02:27:44.922490 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.922479    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" containerName="copy"
May 29 02:27:44.922536 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.922491    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" containerName="copy"
May 29 02:27:44.922536 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.922511    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" containerName="gather"
May 29 02:27:44.922536 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.922517    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" containerName="gather"
May 29 02:27:44.922674 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.922556    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" containerName="gather"
May 29 02:27:44.922674 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.922568    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="e5a11bbd-4699-4010-be1e-9ff591918114" containerName="copy"
May 29 02:27:44.927322 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.927302    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:44.929555 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.929528    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bphng\"/\"kube-root-ca.crt\""
May 29 02:27:44.929688 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.929615    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bphng\"/\"openshift-service-ca.crt\""
May 29 02:27:44.930077 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.930063    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bphng\"/\"default-dockercfg-5v6cc\""
May 29 02:27:44.934485 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.934197    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"]
May 29 02:27:44.993511 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.993473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-sys\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:44.993722 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.993543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-proc\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:44.993722 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.993571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-lib-modules\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:44.993722 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.993624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5jk\" (UniqueName: \"kubernetes.io/projected/1c6f1e79-f99f-48be-a7cc-3a441caee977-kube-api-access-jp5jk\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:44.993722 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:44.993669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-podres\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.094089 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.094057    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-sys\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.094089 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.094092    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-proc\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.094310 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.094115    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-lib-modules\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.094310 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.094132    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5jk\" (UniqueName: \"kubernetes.io/projected/1c6f1e79-f99f-48be-a7cc-3a441caee977-kube-api-access-jp5jk\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.094310 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.094161    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-podres\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.094310 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.094189    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-sys\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.094310 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.094189    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-proc\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.094310 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.094260    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-podres\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.094310 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.094275    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c6f1e79-f99f-48be-a7cc-3a441caee977-lib-modules\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.101880 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.101823    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5jk\" (UniqueName: \"kubernetes.io/projected/1c6f1e79-f99f-48be-a7cc-3a441caee977-kube-api-access-jp5jk\") pod \"perf-node-gather-daemonset-l8ttm\" (UID: \"1c6f1e79-f99f-48be-a7cc-3a441caee977\") " pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.238850 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.238805    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:45.363633 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.361351    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"]
May 29 02:27:45.367916 ip-10-0-129-62 kubenswrapper[2580]: W0529 02:27:45.367887    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c6f1e79_f99f_48be_a7cc_3a441caee977.slice/crio-0a54223b9505d103e09cfc69be7b6e1d4a2d96330a6ccbf0ac6eec779c3edcfc WatchSource:0}: Error finding container 0a54223b9505d103e09cfc69be7b6e1d4a2d96330a6ccbf0ac6eec779c3edcfc: Status 404 returned error can't find the container with id 0a54223b9505d103e09cfc69be7b6e1d4a2d96330a6ccbf0ac6eec779c3edcfc
May 29 02:27:45.369454 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.369434    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 02:27:45.408613 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.408567    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm" event={"ID":"1c6f1e79-f99f-48be-a7cc-3a441caee977","Type":"ContainerStarted","Data":"0a54223b9505d103e09cfc69be7b6e1d4a2d96330a6ccbf0ac6eec779c3edcfc"}
May 29 02:27:45.497153 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.497111    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fqm6l_2cd392d5-8760-45f1-9164-4d9c714a6de3/dns/0.log"
May 29 02:27:45.514342 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.514308    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fqm6l_2cd392d5-8760-45f1-9164-4d9c714a6de3/kube-rbac-proxy/0.log"
May 29 02:27:45.615989 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:45.615914    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kdkn5_ae3f09b5-807b-46e2-a6ea-cb7d47ae16db/dns-node-resolver/0.log"
May 29 02:27:46.070538 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:46.070511    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lmdcx_41a9f0b2-1f52-4fb2-b920-617ce22dc8d0/node-ca/0.log"
May 29 02:27:46.412899 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:46.412813    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm" event={"ID":"1c6f1e79-f99f-48be-a7cc-3a441caee977","Type":"ContainerStarted","Data":"dfcb5a14c069c47bd95f1e9c0945a01e890ba1860714e3a6daf72715baf87d84"}
May 29 02:27:46.413260 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:46.412920    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:46.430415 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:46.430363    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm" podStartSLOduration=2.430343183 podStartE2EDuration="2.430343183s" podCreationTimestamp="2026-05-29 02:27:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:27:46.428987608 +0000 UTC m=+1763.011978523" watchObservedRunningTime="2026-05-29 02:27:46.430343183 +0000 UTC m=+1763.013334088"
May 29 02:27:46.732844 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:46.732764    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b7cd4d9b6-v4s84_be7b6307-7544-459b-a527-e69e07916e7a/router/0.log"
May 29 02:27:47.098453 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:47.098425    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lbjht_d21fba59-d15e-4394-bf25-23ba74c209e4/serve-healthcheck-canary/0.log"
May 29 02:27:47.550487 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:47.550458    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gqztr_d0246d78-bf31-4061-ab18-610154dabef5/kube-rbac-proxy/0.log"
May 29 02:27:47.569177 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:47.569142    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gqztr_d0246d78-bf31-4061-ab18-610154dabef5/exporter/0.log"
May 29 02:27:47.586652 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:47.586631    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gqztr_d0246d78-bf31-4061-ab18-610154dabef5/extractor/0.log"
May 29 02:27:51.696814 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:51.696780    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-cgp2h_447123ce-921a-49f9-8634-fbcc2d3e0457/migrator/0.log"
May 29 02:27:51.715867 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:51.715845    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-cgp2h_447123ce-921a-49f9-8634-fbcc2d3e0457/graceful-termination/0.log"
May 29 02:27:51.977283 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:51.977194    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-h7sph_37947119-0606-40c1-9ea3-eb3005ce71e3/kube-storage-version-migrator-operator/1.log"
May 29 02:27:51.978064 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:51.978044    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-h7sph_37947119-0606-40c1-9ea3-eb3005ce71e3/kube-storage-version-migrator-operator/0.log"
May 29 02:27:52.424824 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:52.424800    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bphng/perf-node-gather-daemonset-l8ttm"
May 29 02:27:52.880608 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:52.880564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p47b_06088ba1-03c3-493f-9665-13dfd7b77b04/kube-multus-additional-cni-plugins/0.log"
May 29 02:27:52.898302 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:52.898274    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p47b_06088ba1-03c3-493f-9665-13dfd7b77b04/egress-router-binary-copy/0.log"
May 29 02:27:52.916872 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:52.916852    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p47b_06088ba1-03c3-493f-9665-13dfd7b77b04/cni-plugins/0.log"
May 29 02:27:52.934358 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:52.934338    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p47b_06088ba1-03c3-493f-9665-13dfd7b77b04/bond-cni-plugin/0.log"
May 29 02:27:52.955863 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:52.955839    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p47b_06088ba1-03c3-493f-9665-13dfd7b77b04/routeoverride-cni/0.log"
May 29 02:27:52.973326 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:52.973303    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p47b_06088ba1-03c3-493f-9665-13dfd7b77b04/whereabouts-cni-bincopy/0.log"
May 29 02:27:52.990868 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:52.990845    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9p47b_06088ba1-03c3-493f-9665-13dfd7b77b04/whereabouts-cni/0.log"
May 29 02:27:53.179163 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:53.179091    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mt97d_2f9bfc3c-879e-4385-8c93-729bd0bac31d/kube-multus/0.log"
May 29 02:27:53.255573 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:53.255543    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7475l_7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9/network-metrics-daemon/0.log"
May 29 02:27:53.273545 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:53.273522    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7475l_7c3a7bb8-d8a7-475a-99dc-d8ad9a10bcc9/kube-rbac-proxy/0.log"
May 29 02:27:54.778258 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:54.778185    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-controller/0.log"
May 29 02:27:54.794067 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:54.794043    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/0.log"
May 29 02:27:54.801164 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:54.801146    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovn-acl-logging/1.log"
May 29 02:27:54.818746 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:54.818725    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/kube-rbac-proxy-node/0.log"
May 29 02:27:54.839872 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:54.839850    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/kube-rbac-proxy-ovn-metrics/0.log"
May 29 02:27:54.856090 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:54.856066    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/northd/0.log"
May 29 02:27:54.872081 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:54.872066    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/nbdb/0.log"
May 29 02:27:54.888876 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:54.888858    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/sbdb/0.log"
May 29 02:27:54.983850 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:54.983821    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pktj7_e064a153-5596-4b6e-9de1-e7734036edc9/ovnkube-controller/0.log"
May 29 02:27:55.957765 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:55.957735    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qfhm9_734ad269-957d-44e9-9400-4270c2f494ae/network-check-target-container/0.log"
May 29 02:27:56.811298 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:56.811271    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8trgn_8a44c991-a799-4094-8818-44f1617948e4/iptables-alerter/0.log"
May 29 02:27:57.371918 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:57.371889    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-hj87r_79261d39-6534-47f2-9d07-2e62121b5e6d/tuned/0.log"
May 29 02:27:59.060361 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:59.060317    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-t5sb7_8d030b6a-eb72-47d2-8e07-7f8146110a99/cluster-samples-operator/0.log"
May 29 02:27:59.074391 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:59.074369    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-t5sb7_8d030b6a-eb72-47d2-8e07-7f8146110a99/cluster-samples-operator-watch/0.log"
May 29 02:27:59.957330 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:59.957298    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-t6jld_eedde3d0-971e-40bf-97f8-5c21637f916f/service-ca-operator/1.log"
May 29 02:27:59.958245 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:27:59.958226    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-t6jld_eedde3d0-971e-40bf-97f8-5c21637f916f/service-ca-operator/0.log"
May 29 02:28:00.733733 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:28:00.733707    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-q7k57_b85086f6-c65d-4db3-ba11-63438c9c34e4/csi-driver/0.log"
May 29 02:28:00.749154 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:28:00.749134    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-q7k57_b85086f6-c65d-4db3-ba11-63438c9c34e4/csi-node-driver-registrar/0.log"
May 29 02:28:00.765940 ip-10-0-129-62 kubenswrapper[2580]: I0529 02:28:00.765917    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-q7k57_b85086f6-c65d-4db3-ba11-63438c9c34e4/csi-liveness-probe/0.log"