May 28 00:11:30.543916 ip-10-0-142-230 systemd[1]: Starting Kubernetes Kubelet...
May 28 00:11:31.008069 ip-10-0-142-230 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 00:11:31.008069 ip-10-0-142-230 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 28 00:11:31.008069 ip-10-0-142-230 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 00:11:31.008069 ip-10-0-142-230 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 28 00:11:31.008069 ip-10-0-142-230 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 00:11:31.009902 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.009819    2560 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 28 00:11:31.017424 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017403    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:31.017424 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017419    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:31.017424 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017424    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:31.017424 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017428    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:31.017424 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017431    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017434    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017437    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017439    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017442    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017445    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017448    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017450    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017453    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017455    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017458    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017461    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017463    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017466    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017468    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017471    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017473    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017476    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017478    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:31.017685 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017481    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017492    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017496    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017499    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017502    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017505    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017507    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017510    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017528    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017530    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017533    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017536    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017540    2560 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017543    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017546    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017549    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017552    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017555    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017557    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:31.018129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017560    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017563    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017565    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017568    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017570    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017573    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017576    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017578    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017581    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017584    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017586    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017589    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017591    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017594    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017596    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017598    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017601    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017603    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017606    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017608    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:31.018635 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017610    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017613    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017616    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017620    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017623    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017626    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017629    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017631    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017636    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017638    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017641    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017643    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017646    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017648    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017651    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017653    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017657    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017660    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017662    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:31.019110 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017665    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017667    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017670    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017672    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.017675    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018049    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018054    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018057    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018060    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018063    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018065    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018068    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018071    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018073    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018076    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018078    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018081    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018083    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018086    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018088    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:31.019580 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018090    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018094    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018097    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018100    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018102    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018104    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018107    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018110    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018112    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018115    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018118    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018120    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018123    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018125    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018128    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018130    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018132    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018135    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018137    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018140    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:31.020047 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018142    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018145    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018147    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018149    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018152    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018155    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018157    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018160    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018163    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018165    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018168    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018171    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018173    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018176    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018178    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018181    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018183    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018186    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018188    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018190    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:31.020570 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018193    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018197    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018201    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018204    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018208    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018211    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018214    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018216    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018219    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018221    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018223    2560 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018226    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018229    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018231    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018235    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018237    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018239    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018243    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018246    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:31.021062 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018248    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018251    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018253    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018256    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018258    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018260    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018263    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018265    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018267    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018270    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018273    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018275    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018345    2560 flags.go:64] FLAG: --address="0.0.0.0"
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018352    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018359    2560 flags.go:64] FLAG: --anonymous-auth="true"
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018364    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018368    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018371    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018376    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018389    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018392    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 28 00:11:31.021612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018395    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018399    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018402    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018405    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018408    2560 flags.go:64] FLAG: --cgroup-root=""
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018411    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018414    2560 flags.go:64] FLAG: --client-ca-file=""
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018417    2560 flags.go:64] FLAG: --cloud-config=""
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018419    2560 flags.go:64] FLAG: --cloud-provider="external"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018426    2560 flags.go:64] FLAG: --cluster-dns="[]"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018430    2560 flags.go:64] FLAG: --cluster-domain=""
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018432    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018435    2560 flags.go:64] FLAG: --config-dir=""
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018438    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018441    2560 flags.go:64] FLAG: --container-log-max-files="5"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018445    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018448    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018451    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018454    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018457    2560 flags.go:64] FLAG: --contention-profiling="false"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018460    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018462    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018465    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018468    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018472    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 28 00:11:31.022110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018475    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018478    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018480    2560 flags.go:64] FLAG: --enable-load-reader="false"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018484    2560 flags.go:64] FLAG: --enable-server="true"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018487    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018492    2560 flags.go:64] FLAG: --event-burst="100"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018495    2560 flags.go:64] FLAG: --event-qps="50"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018497    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018501    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018503    2560 flags.go:64] FLAG: --eviction-hard=""
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018526    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018530    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018532    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018536    2560 flags.go:64] FLAG: --eviction-soft=""
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018539    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018541    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018546    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018549    2560 flags.go:64] FLAG: --experimental-mounter-path=""
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018552    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018555    2560 flags.go:64] FLAG: --fail-swap-on="true"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018558    2560 flags.go:64] FLAG: --feature-gates=""
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018561    2560 flags.go:64] FLAG: --file-check-frequency="20s"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018564    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018567    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018570    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018574    2560 flags.go:64] FLAG: --healthz-port="10248"
May 28 00:11:31.022721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018577    2560 flags.go:64] FLAG: --help="false"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018580    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-142-230.ec2.internal"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018583    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018586    2560 flags.go:64] FLAG: --http-check-frequency="20s"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018589    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018592    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018595    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018598    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018601    2560 flags.go:64] FLAG: --image-service-endpoint=""
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018604    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018607    2560 flags.go:64] FLAG: --kube-api-burst="100"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018610    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018613    2560 flags.go:64] FLAG: --kube-api-qps="50"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018616    2560 flags.go:64] FLAG: --kube-reserved=""
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018619    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018622    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018625    2560 flags.go:64] FLAG: --kubelet-cgroups=""
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018628    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018631    2560 flags.go:64] FLAG: --lock-file=""
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018634    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018637    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018639    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018646    2560 flags.go:64] FLAG: --log-json-split-stream="false"
May 28 00:11:31.023357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018649    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018652    2560 flags.go:64] FLAG: --log-text-split-stream="false"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018654    2560 flags.go:64] FLAG: --logging-format="text"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018657    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018660    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018663    2560 flags.go:64] FLAG: --manifest-url=""
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018666    2560 flags.go:64] FLAG: --manifest-url-header=""
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018670    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018673    2560 flags.go:64] FLAG: --max-open-files="1000000"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018677    2560 flags.go:64] FLAG: --max-pods="110"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018680    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018683    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018685    2560 flags.go:64] FLAG: --memory-manager-policy="None"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018688    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018691    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018693    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018696    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018703    2560 flags.go:64] FLAG: --node-status-max-images="50"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018706    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018709    2560 flags.go:64] FLAG: --oom-score-adj="-999"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018712    2560 flags.go:64] FLAG: --pod-cidr=""
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018715    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018720    2560 flags.go:64] FLAG: --pod-manifest-path=""
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018723    2560 flags.go:64] FLAG: --pod-max-pids="-1"
May 28 00:11:31.023915 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018727    2560 flags.go:64] FLAG: --pods-per-core="0"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018729    2560 flags.go:64] FLAG: --port="10250"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018732    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018736    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b9b0107cb2ed3545"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018739    2560 flags.go:64] FLAG: --qos-reserved=""
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018741    2560 flags.go:64] FLAG: --read-only-port="10255"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018744    2560 flags.go:64] FLAG: --register-node="true"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018747    2560 flags.go:64] FLAG: --register-schedulable="true"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018751    2560 flags.go:64] FLAG: --register-with-taints=""
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018755    2560 flags.go:64] FLAG: --registry-burst="10"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018757    2560 flags.go:64] FLAG: --registry-qps="5"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018760    2560 flags.go:64] FLAG: --reserved-cpus=""
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018763    2560 flags.go:64] FLAG: --reserved-memory=""
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018766    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018769    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018772    2560 flags.go:64] FLAG: --rotate-certificates="false"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018775    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018778    2560 flags.go:64] FLAG: --runonce="false"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018780    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018784    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018786    2560 flags.go:64] FLAG: --seccomp-default="false"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018789    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018792    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018795    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018798    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018800    2560 flags.go:64] FLAG: --storage-driver-password="root"
May 28 00:11:31.024485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018803    2560 flags.go:64] FLAG: --storage-driver-secure="false"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018806    2560 flags.go:64] FLAG: --storage-driver-table="stats"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018808    2560 flags.go:64] FLAG: --storage-driver-user="root"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018811    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018814    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018817    2560 flags.go:64] FLAG: --system-cgroups=""
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018820    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018825    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018828    2560 flags.go:64] FLAG: --tls-cert-file=""
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018831    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018834    2560 flags.go:64] FLAG: --tls-min-version=""
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018837    2560 flags.go:64] FLAG: --tls-private-key-file=""
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018840    2560 flags.go:64] FLAG: --topology-manager-policy="none"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018843    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018846    2560 flags.go:64] FLAG: --topology-manager-scope="container"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018849    2560 flags.go:64] FLAG: --v="2"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018853    2560 flags.go:64] FLAG: --version="false"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018857    2560 flags.go:64] FLAG: --vmodule=""
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018861    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.018864    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018960    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018963    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018966    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018968    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:31.025227 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018971    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018973    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018976    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018978    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018981    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018984    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018987    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018989    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018991    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018994    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018996    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.018999    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019002    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019005    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019007    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019010    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019012    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019014    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019017    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019019    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:31.025832 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019022    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019025    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019029    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019031    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019034    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019036    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019039    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019041    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019045    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019047    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019049    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019052    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019054    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019057    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019059    2560 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019061    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019064    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019066    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019069    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:31.026330 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019071    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019073    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019076    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019078    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019081    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019083    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019086    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019088    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019091    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019093    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019095    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019098    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019100    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019103    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019106    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019111    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019115    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019119    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019123    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:31.026820 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019125    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019128    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019132    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019134    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019137    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019139    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019142    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019144    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019146    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019149    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019151    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019153    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019156    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019158    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019160    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019163    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019165    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019167    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019170    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019172    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:31.027273 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019175    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019178    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019180    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.019182    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.019978    2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.026424    2560 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.026441    2560 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026490    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026495    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026499    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026501    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026504    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026507    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026510    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026532    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026535    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:31.027807 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026538    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026541    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026544    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026546    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026549    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026552    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026554    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026557    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026560    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026563    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026565    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026567    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026570    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026572    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026575    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026577    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026579    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026582    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026584    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026587    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:31.028197 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026590    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026593    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026603    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026605    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026608    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026610    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026613    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026615    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026618    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026620    2560 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026623    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026625    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026628    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026630    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026633    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026636    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026639    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026641    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026644    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:31.028730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026646    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026649    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026651    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026653    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026656    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026658    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026661    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026663    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026665    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026668    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026670    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026673    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026675    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026678    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026681    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026683    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026692    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026694    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026697    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026699    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:31.029202 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026701    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026704    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026706    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026710    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026715    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026718    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026721    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026724    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026726    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026730    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026733    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026736    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026738    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026740    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026743    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026745    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026747    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:31.029721 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026750    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.026755    2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026861    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026867    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026869    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026872    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026875    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026878    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026880    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026883    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026886    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026889    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026892    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026894    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026897    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:31.030356 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026899    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026902    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026904    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026908    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026911    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026913    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026916    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026918    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026920    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026923    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026925    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026928    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026930    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026933    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026935    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026938    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026940    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026943    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026945    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:31.030764 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026948    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026950    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026953    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026957    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026960    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026963    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026966    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026969    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026972    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026975    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026977    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026980    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026983    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026986    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026988    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026991    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026993    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026995    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.026998    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:31.031300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027001    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027003    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027006    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027008    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027010    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027013    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027015    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027018    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027020    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027022    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027025    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027027    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027030    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027032    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027035    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027037    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027040    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027042    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027045    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027047    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:31.031936 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027050    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027052    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027055    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027057    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027060    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027063    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027065    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027068    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027070    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027073    2560 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027075    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027077    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027080    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027082    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:31.027085    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.027089    2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 00:11:31.032461 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.027782    2560 server.go:962] "Client rotation is on, will bootstrap in background"
May 28 00:11:31.032889 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.029754    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 28 00:11:31.032889 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.030802    2560 server.go:1019] "Starting client certificate rotation"
May 28 00:11:31.032889 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.030920    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 00:11:31.032889 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.030970    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 00:11:31.056252 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.056229    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 00:11:31.073922 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.073888    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 00:11:31.084930 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.084910    2560 log.go:25] "Validated CRI v1 runtime API"
May 28 00:11:31.090086 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.090071    2560 log.go:25] "Validated CRI v1 image API"
May 28 00:11:31.091402 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.091383    2560 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 28 00:11:31.094070 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.094047    2560 fs.go:135] Filesystem UUIDs: map[1ac891bc-acef-4ec2-8636-8fa5851ba3a6:/dev/nvme0n1p4 594b8c7d-586d-42d9-b4d6-1708a44df121:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
May 28 00:11:31.094150 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.094069    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 28 00:11:31.094448 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.094429    2560 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 00:11:31.098462 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.098340    2560 manager.go:217] Machine: {Timestamp:2026-05-28 00:11:31.097850188 +0000 UTC m=+0.425516287 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3033588 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec287449b98be42aa371ebd074247c27 SystemUUID:ec287449-b98b-e42a-a371-ebd074247c27 BootID:3ac791d4-0d05-4ef8-af60-dfa87698be7f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:11:59:5c:d8:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:11:59:5c:d8:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:42:43:46:49:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 28 00:11:31.098462 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.098449    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 28 00:11:31.098649 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.098572    2560 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 28 00:11:31.100420 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.100398    2560 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 28 00:11:31.100571 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.100422    2560 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-230.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 28 00:11:31.101067 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.101056    2560 topology_manager.go:138] "Creating topology manager with none policy"
May 28 00:11:31.101098 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.101070    2560 container_manager_linux.go:306] "Creating device plugin manager"
May 28 00:11:31.101098 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.101078    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 00:11:31.102225 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.102214    2560 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 00:11:31.103374 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.103364    2560 state_mem.go:36] "Initialized new in-memory state store"
May 28 00:11:31.103663 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.103654    2560 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 28 00:11:31.106050 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.106041    2560 kubelet.go:491] "Attempting to sync node with API server"
May 28 00:11:31.106089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.106059    2560 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 28 00:11:31.106089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.106071    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 28 00:11:31.106089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.106080    2560 kubelet.go:397] "Adding apiserver pod source"
May 28 00:11:31.106089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.106090    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 28 00:11:31.107184 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.107173    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 00:11:31.107218 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.107191    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 00:11:31.110581 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.110563    2560 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 28 00:11:31.112284 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.112271    2560 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 28 00:11:31.113161 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113149    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 28 00:11:31.113198 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113170    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 28 00:11:31.113198 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113179    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 28 00:11:31.113198 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113184    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 28 00:11:31.113198 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113192    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 28 00:11:31.113198 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113197    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 28 00:11:31.113327 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113203    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 28 00:11:31.113327 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113208    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 28 00:11:31.113327 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113215    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 28 00:11:31.113327 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113221    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 28 00:11:31.113327 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113238    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 28 00:11:31.113575 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.113565    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 28 00:11:31.114415 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.114407    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 28 00:11:31.114453 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.114425    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 28 00:11:31.117099 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.117074    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9st4v"
May 28 00:11:31.117581 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.117562    2560 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-230.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 28 00:11:31.117892 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.117829    2560 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 28 00:11:31.118135 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.118109    2560 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 28 00:11:31.118403 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.118389    2560 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 28 00:11:31.118458 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.118435    2560 server.go:1295] "Started kubelet"
May 28 00:11:31.118577 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.118530    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 28 00:11:31.118640 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.118553    2560 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 28 00:11:31.118640 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.118613    2560 server_v1.go:47] "podresources" method="list" useActivePods=true
May 28 00:11:31.119280 ip-10-0-142-230 systemd[1]: Started Kubernetes Kubelet.
May 28 00:11:31.119861 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.119682    2560 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 28 00:11:31.121101 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.121089    2560 server.go:317] "Adding debug handlers to kubelet server"
May 28 00:11:31.124116 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.124100    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9st4v"
May 28 00:11:31.129033 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.122781    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-230.ec2.internal.18b392424834e8b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-230.ec2.internal,UID:ip-10-0-142-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-230.ec2.internal,},FirstTimestamp:2026-05-28 00:11:31.118401713 +0000 UTC m=+0.446067812,LastTimestamp:2026-05-28 00:11:31.118401713 +0000 UTC m=+0.446067812,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-230.ec2.internal,}"
May 28 00:11:31.132143 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.132124    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 28 00:11:31.132753 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.132735    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 28 00:11:31.132840 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.132822    2560 volume_manager.go:295] "The desired_state_of_world populator starts"
May 28 00:11:31.132840 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.132834    2560 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 28 00:11:31.132944 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.132887    2560 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 28 00:11:31.132944 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.132928    2560 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 28 00:11:31.133044 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.132976    2560 reconstruct.go:97] "Volume reconstruction finished"
May 28 00:11:31.133044 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.132983    2560 reconciler.go:26] "Reconciler: start to sync state"
May 28 00:11:31.133615 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.133596    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:31.134251 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.134235    2560 factory.go:153] Registering CRI-O factory
May 28 00:11:31.134307 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.134298    2560 factory.go:223] Registration of the crio container factory successfully
May 28 00:11:31.134364 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.134353    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 28 00:11:31.134412 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.134367    2560 factory.go:55] Registering systemd factory
May 28 00:11:31.134412 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.134375    2560 factory.go:223] Registration of the systemd container factory successfully
May 28 00:11:31.134412 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.134389    2560 factory.go:103] Registering Raw factory
May 28 00:11:31.134412 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.134401    2560 manager.go:1196] Started watching for new ooms in manager
May 28 00:11:31.134766 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.134753    2560 manager.go:319] Starting recovery of all containers
May 28 00:11:31.135263 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.135241    2560 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 28 00:11:31.135335 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.135288    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 28 00:11:31.146000 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.145838    2560 manager.go:324] Recovery completed
May 28 00:11:31.150084 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.150072    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:31.157137 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.157121    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:31.157198 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.157149    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:31.157198 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.157159    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:31.157598 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.157583    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 28 00:11:31.157598 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.157597    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 28 00:11:31.157699 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.157614    2560 state_mem.go:36] "Initialized new in-memory state store"
May 28 00:11:31.159889 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.159877    2560 policy_none.go:49] "None policy: Start"
May 28 00:11:31.159931 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.159894    2560 memory_manager.go:186] "Starting memorymanager" policy="None"
May 28 00:11:31.159931 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.159904    2560 state_mem.go:35] "Initializing new in-memory state store"
May 28 00:11:31.203044 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.203030    2560 manager.go:341] "Starting Device Plugin manager"
May 28 00:11:31.203159 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.203061    2560 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 28 00:11:31.203159 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.203074    2560 server.go:85] "Starting device plugin registration server"
May 28 00:11:31.203305 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.203293    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
May 28 00:11:31.203364 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.203307    2560 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 28 00:11:31.203412 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.203385    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 28 00:11:31.203459 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.203453    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 28 00:11:31.203504 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.203461    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 28 00:11:31.203963 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.203944    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 28 00:11:31.204048 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.203986    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:31.254571 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.254549    2560 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 28 00:11:31.255862 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.255846    2560 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 28 00:11:31.255945 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.255866    2560 status_manager.go:230] "Starting to sync pod status with apiserver"
May 28 00:11:31.255945 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.255881    2560 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 28 00:11:31.255945 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.255887    2560 kubelet.go:2451] "Starting kubelet main sync loop"
May 28 00:11:31.255945 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.255924    2560 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 28 00:11:31.261182 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.261134    2560 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:31.304304 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.304281    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:31.304986 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.304963    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:31.305068 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.304998    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:31.305068 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.305009    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:31.305068 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.305031    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-230.ec2.internal"
May 28 00:11:31.312413 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.312400    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-230.ec2.internal"
May 28 00:11:31.312453 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.312420    2560 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-230.ec2.internal\": node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:31.324078 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.324056    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:31.356260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.356241    2560 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"]
May 28 00:11:31.356313 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.356299    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:31.357016 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.357002    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:31.357079 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.357028    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:31.357079 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.357039    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:31.358173 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.358161    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:31.358361 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.358346    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.358402 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.358378    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:31.358851 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.358836    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:31.358927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.358836    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:31.358927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.358887    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:31.358927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.358908    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:31.359013 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.358862    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:31.359013 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.358945    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:31.360071 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.360054    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.360134 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.360080    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:31.360705 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.360688    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:31.360705 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.360716    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:31.360833 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.360726    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:31.390047 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.390025    2560 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-230.ec2.internal\" not found" node="ip-10-0-142-230.ec2.internal"
May 28 00:11:31.394293 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.394279    2560 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-230.ec2.internal\" not found" node="ip-10-0-142-230.ec2.internal"
May 28 00:11:31.424244 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.424224    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:31.434981 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.434961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0471fde4bca3037bb39fa531a41ff217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"0471fde4bca3037bb39fa531a41ff217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.435051 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.434985    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0471fde4bca3037bb39fa531a41ff217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"0471fde4bca3037bb39fa531a41ff217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.435051 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.435002    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6fac2f13116ec505f7f49d6272df8583-config\") pod \"kube-apiserver-proxy-ip-10-0-142-230.ec2.internal\" (UID: \"6fac2f13116ec505f7f49d6272df8583\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.525365 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.525314    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:31.535311 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.535294    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0471fde4bca3037bb39fa531a41ff217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"0471fde4bca3037bb39fa531a41ff217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.535362 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.535317    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0471fde4bca3037bb39fa531a41ff217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"0471fde4bca3037bb39fa531a41ff217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.535362 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.535335    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6fac2f13116ec505f7f49d6272df8583-config\") pod \"kube-apiserver-proxy-ip-10-0-142-230.ec2.internal\" (UID: \"6fac2f13116ec505f7f49d6272df8583\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.535433 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.535366    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6fac2f13116ec505f7f49d6272df8583-config\") pod \"kube-apiserver-proxy-ip-10-0-142-230.ec2.internal\" (UID: \"6fac2f13116ec505f7f49d6272df8583\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.535433 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.535387    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0471fde4bca3037bb39fa531a41ff217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"0471fde4bca3037bb39fa531a41ff217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.535433 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.535387    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0471fde4bca3037bb39fa531a41ff217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"0471fde4bca3037bb39fa531a41ff217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.625714 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.625687    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:31.693144 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.693125    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.696830 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:31.696805    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
May 28 00:11:31.726270 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.726249    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:31.826701 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.826670    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:31.927146 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:31.927123    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:32.027550 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:32.027511    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:32.030651 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.030621    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 28 00:11:32.030767 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.030753    2560 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 00:11:32.128430 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:32.128406    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:32.131330 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.131300    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-27 00:06:31 +0000 UTC" deadline="2028-02-11 12:09:37.503294554 +0000 UTC"
May 28 00:11:32.131330 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.131328    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14987h58m5.371970278s"
May 28 00:11:32.132400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.132384    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 28 00:11:32.139596 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.139578    2560 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 00:11:32.161717 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.161692    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7bvkq"
May 28 00:11:32.173660 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.173637    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7bvkq"
May 28 00:11:32.175600 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:32.175580    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0471fde4bca3037bb39fa531a41ff217.slice/crio-b7fd948cb346506cc7503bcb76e470c797c879d6c79a3227a659dac25503d4ff WatchSource:0}: Error finding container b7fd948cb346506cc7503bcb76e470c797c879d6c79a3227a659dac25503d4ff: Status 404 returned error can't find the container with id b7fd948cb346506cc7503bcb76e470c797c879d6c79a3227a659dac25503d4ff
May 28 00:11:32.175730 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:32.175717    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fac2f13116ec505f7f49d6272df8583.slice/crio-e68987dbba8ed906a9fe20968ed784db2109d07b689912b4fd139a56fd5c6e81 WatchSource:0}: Error finding container e68987dbba8ed906a9fe20968ed784db2109d07b689912b4fd139a56fd5c6e81: Status 404 returned error can't find the container with id e68987dbba8ed906a9fe20968ed784db2109d07b689912b4fd139a56fd5c6e81
May 28 00:11:32.180473 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.180458    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:11:32.188327 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.188306    2560 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:32.229290 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:32.229269    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
May 28 00:11:32.258372 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.258300    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal" event={"ID":"0471fde4bca3037bb39fa531a41ff217","Type":"ContainerStarted","Data":"b7fd948cb346506cc7503bcb76e470c797c879d6c79a3227a659dac25503d4ff"}
May 28 00:11:32.259129 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.259108    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal" event={"ID":"6fac2f13116ec505f7f49d6272df8583","Type":"ContainerStarted","Data":"e68987dbba8ed906a9fe20968ed784db2109d07b689912b4fd139a56fd5c6e81"}
May 28 00:11:32.260415 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.260402    2560 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:32.305472 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.305453    2560 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:32.333849 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.333797    2560 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
May 28 00:11:32.342064 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.342045    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 00:11:32.343062 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.343051    2560 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
May 28 00:11:32.352883 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:32.352864    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 00:11:33.107875 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.107846    2560 apiserver.go:52] "Watching apiserver"
May 28 00:11:33.112985 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.112958    2560 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 28 00:11:33.114106 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.114075    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal","openshift-cluster-node-tuning-operator/tuned-f5hsm","openshift-image-registry/node-ca-qfqhg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal","openshift-multus/multus-additional-cni-plugins-tk77g","openshift-multus/network-metrics-daemon-2dnl7","openshift-network-diagnostics/network-check-target-4sgrj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg","openshift-dns/node-resolver-7qxzh","openshift-multus/multus-59tbq","openshift-network-operator/iptables-alerter-crd6s","openshift-ovn-kubernetes/ovnkube-node-t4zhg","kube-system/konnectivity-agent-45gzz"]
May 28 00:11:33.116379 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.116358    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-59tbq"
May 28 00:11:33.117543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.117413    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.117543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.117531    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.118424 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.118401    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 28 00:11:33.118533 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.118441    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 28 00:11:33.118533 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.118496    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmq8c\""
May 28 00:11:33.118621 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.118448    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 28 00:11:33.118621 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.118604    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 28 00:11:33.119238 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.119210    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.119339 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.119260    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 28 00:11:33.119339 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.119238    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 28 00:11:33.119469 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.119415    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 28 00:11:33.120542 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.119687    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 28 00:11:33.120542 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.119836    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 28 00:11:33.120542 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.120114    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7nqc7\""
May 28 00:11:33.120542 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.120252    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9bg6w\""
May 28 00:11:33.120825 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.120815    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 28 00:11:33.121073 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.121054    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xm8tm\""
May 28 00:11:33.121214 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.121200    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 28 00:11:33.121344 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.121328    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:33.121452 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.121404    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:33.121593 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.121452    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:33.121593 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.121532    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:33.123371 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.123352    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.125290 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.125273    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 28 00:11:33.125396 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.125379    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zswfr\""
May 28 00:11:33.125707 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.125690    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 28 00:11:33.125807 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.125736    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 28 00:11:33.127625 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.126570    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.127625 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.126889    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.128226 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.128195    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 28 00:11:33.128317 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.128245    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 28 00:11:33.128375 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.128342    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 28 00:11:33.128375 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.128371    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ktw8b\""
May 28 00:11:33.128609 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.128582    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 28 00:11:33.129098 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.129081    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j9djt\""
May 28 00:11:33.129187 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.129137    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 28 00:11:33.129187 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.129153    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.130354 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.130338    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:33.130802 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.130785    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 28 00:11:33.130946 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.130933    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bj4w5\""
May 28 00:11:33.131322 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.130937    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 28 00:11:33.131322 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.131069    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 28 00:11:33.131322 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.131079    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 28 00:11:33.131322 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.131069    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 28 00:11:33.131322 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.131146    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 28 00:11:33.131938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.131919    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 28 00:11:33.132043 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.132028    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 28 00:11:33.132229 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.132210    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jl9qk\""
May 28 00:11:33.133864 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.133849    2560 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 28 00:11:33.141727 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141709    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbde4ba0-8948-4141-93a0-c0ff1aa62641-hosts-file\") pod \"node-resolver-7qxzh\" (UID: \"bbde4ba0-8948-4141-93a0-c0ff1aa62641\") " pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.141816 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttpp4\" (UniqueName: \"kubernetes.io/projected/bbde4ba0-8948-4141-93a0-c0ff1aa62641-kube-api-access-ttpp4\") pod \"node-resolver-7qxzh\" (UID: \"bbde4ba0-8948-4141-93a0-c0ff1aa62641\") " pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.141816 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-os-release\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.141816 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-system-cni-dir\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.141816 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbde4ba0-8948-4141-93a0-c0ff1aa62641-tmp-dir\") pod \"node-resolver-7qxzh\" (UID: \"bbde4ba0-8948-4141-93a0-c0ff1aa62641\") " pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.141816 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141806    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a052ebe-16e4-40a8-b822-8160f7735c0b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-socket-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141849    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-kubelet\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-run-ovn\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-cni-bin\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-sysctl-d\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-host\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.141985    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/606fba4b-6867-42fe-b598-0dc46cf45c5b-konnectivity-ca\") pod \"konnectivity-agent-45gzz\" (UID: \"606fba4b-6867-42fe-b598-0dc46cf45c5b\") " pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-system-cni-dir\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6hv6\" (UniqueName: \"kubernetes.io/projected/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-kube-api-access-r6hv6\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:33.142069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c9057d1-fd80-4b1c-af89-acfff9aec573-ovnkube-config\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142133    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-cni-netd\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-var-lib-kubelet\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-run-ovn-kubernetes\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c9057d1-fd80-4b1c-af89-acfff9aec573-ovnkube-script-lib\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-var-lib-cni-bin\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-etc-kubernetes\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp2qs\" (UniqueName: \"kubernetes.io/projected/0a052ebe-16e4-40a8-b822-8160f7735c0b-kube-api-access-vp2qs\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-run-netns\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-run-netns\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142416    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9315e6c2-c11e-4ee7-a5fd-5f57e12772a6-serviceca\") pod \"node-ca-qfqhg\" (UID: \"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6\") " pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.142558 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-registration-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-log-socket\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-run-multus-certs\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142537    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-sysconfig\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-sysctl-conf\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-device-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142602    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-socket-dir-parent\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-run-k8s-cni-cncf-io\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-var-lib-cni-multus\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a8254b7-7a5c-4f77-9bf1-64605de1f147-cni-binary-copy\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/777a4765-6b60-4801-95b7-f632242323c4-etc-tuned\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-etc-selinux\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6d96c6b-2765-4f69-934b-00641848c010-host-slash\") pod \"iptables-alerter-crd6s\" (UID: \"e6d96c6b-2765-4f69-934b-00641848c010\") " pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-systemd-units\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-daemon-config\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142870    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psd47\" (UniqueName: \"kubernetes.io/projected/1a8254b7-7a5c-4f77-9bf1-64605de1f147-kube-api-access-psd47\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.143303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bllx\" (UniqueName: \"kubernetes.io/projected/777a4765-6b60-4801-95b7-f632242323c4-kube-api-access-2bllx\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnk5t\" (UniqueName: \"kubernetes.io/projected/3c9057d1-fd80-4b1c-af89-acfff9aec573-kube-api-access-nnk5t\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-modprobe-d\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-systemd\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.142995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/777a4765-6b60-4801-95b7-f632242323c4-tmp\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e6d96c6b-2765-4f69-934b-00641848c010-iptables-alerter-script\") pod \"iptables-alerter-crd6s\" (UID: \"e6d96c6b-2765-4f69-934b-00641848c010\") " pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-slash\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-etc-openvswitch\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-run-openvswitch\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143080    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-conf-dir\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9315e6c2-c11e-4ee7-a5fd-5f57e12772a6-host\") pod \"node-ca-qfqhg\" (UID: \"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6\") " pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58dsl\" (UniqueName: \"kubernetes.io/projected/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-kube-api-access-58dsl\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-run-systemd\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c9057d1-fd80-4b1c-af89-acfff9aec573-ovn-node-metrics-cert\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-hostroot\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-cni-dir\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-os-release\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.144089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-kubernetes\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/606fba4b-6867-42fe-b598-0dc46cf45c5b-agent-certs\") pod \"konnectivity-agent-45gzz\" (UID: \"606fba4b-6867-42fe-b598-0dc46cf45c5b\") " pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143363    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a052ebe-16e4-40a8-b822-8160f7735c0b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-sys-fs\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknvp\" (UniqueName: \"kubernetes.io/projected/e6d96c6b-2765-4f69-934b-00641848c010-kube-api-access-wknvp\") pod \"iptables-alerter-crd6s\" (UID: \"e6d96c6b-2765-4f69-934b-00641848c010\") " pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c9057d1-fd80-4b1c-af89-acfff9aec573-env-overrides\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143469    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-cnibin\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6wx\" (UniqueName: \"kubernetes.io/projected/9315e6c2-c11e-4ee7-a5fd-5f57e12772a6-kube-api-access-df6wx\") pod \"node-ca-qfqhg\" (UID: \"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6\") " pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-node-log\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143541    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-lib-modules\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-cnibin\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a052ebe-16e4-40a8-b822-8160f7735c0b-cni-binary-copy\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143593    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-var-lib-openvswitch\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-run\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-sys\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.144747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.143648    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-var-lib-kubelet\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.174308 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.174280    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 00:06:32 +0000 UTC" deadline="2028-02-07 05:09:20.2353618 +0000 UTC"
May 28 00:11:33.174392 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.174309    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14884h57m47.061055595s"
May 28 00:11:33.243867 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.243837    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-kubelet\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244033 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.243878    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-run-ovn\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244033 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.243901    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-cni-bin\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244033 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.243970    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-kubelet\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244033 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.243975    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-cni-bin\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244033 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244007    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-run-ovn\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244033 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244012    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-sysctl-d\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244039    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-host\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244060    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/606fba4b-6867-42fe-b598-0dc46cf45c5b-konnectivity-ca\") pod \"konnectivity-agent-45gzz\" (UID: \"606fba4b-6867-42fe-b598-0dc46cf45c5b\") " pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244082    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-system-cni-dir\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244102    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hv6\" (UniqueName: \"kubernetes.io/projected/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-kube-api-access-r6hv6\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244127    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244135    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-sysctl-d\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244151    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c9057d1-fd80-4b1c-af89-acfff9aec573-ovnkube-config\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244175    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244191    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-system-cni-dir\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244199    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244229    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-cni-netd\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244234    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-host\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244254    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-var-lib-kubelet\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.244289 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244278    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-run-ovn-kubernetes\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244303    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c9057d1-fd80-4b1c-af89-acfff9aec573-ovnkube-script-lib\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244327    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-var-lib-cni-bin\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244351    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-etc-kubernetes\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244373    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244397    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vp2qs\" (UniqueName: \"kubernetes.io/projected/0a052ebe-16e4-40a8-b822-8160f7735c0b-kube-api-access-vp2qs\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244424    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-run-netns\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244447    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-run-netns\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244471    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9315e6c2-c11e-4ee7-a5fd-5f57e12772a6-serviceca\") pod \"node-ca-qfqhg\" (UID: \"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6\") " pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244497    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244549    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-registration-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244573    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-log-socket\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244597    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-run-multus-certs\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244620    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-sysconfig\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244645    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-sysctl-conf\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244676    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-device-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244700    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-socket-dir-parent\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.244863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244726    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-run-k8s-cni-cncf-io\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244756    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-var-lib-cni-multus\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244765    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/606fba4b-6867-42fe-b598-0dc46cf45c5b-konnectivity-ca\") pod \"konnectivity-agent-45gzz\" (UID: \"606fba4b-6867-42fe-b598-0dc46cf45c5b\") " pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244783    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a8254b7-7a5c-4f77-9bf1-64605de1f147-cni-binary-copy\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244800    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c9057d1-fd80-4b1c-af89-acfff9aec573-ovnkube-config\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244807    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/777a4765-6b60-4801-95b7-f632242323c4-etc-tuned\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244846    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-etc-selinux\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244871    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6d96c6b-2765-4f69-934b-00641848c010-host-slash\") pod \"iptables-alerter-crd6s\" (UID: \"e6d96c6b-2765-4f69-934b-00641848c010\") " pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244894    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-systemd-units\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244921    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-daemon-config\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244947    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-psd47\" (UniqueName: \"kubernetes.io/projected/1a8254b7-7a5c-4f77-9bf1-64605de1f147-kube-api-access-psd47\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244970    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bllx\" (UniqueName: \"kubernetes.io/projected/777a4765-6b60-4801-95b7-f632242323c4-kube-api-access-2bllx\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.244994    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nnk5t\" (UniqueName: \"kubernetes.io/projected/3c9057d1-fd80-4b1c-af89-acfff9aec573-kube-api-access-nnk5t\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245022    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-modprobe-d\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245046    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-systemd\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245070    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/777a4765-6b60-4801-95b7-f632242323c4-tmp\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.245084    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245090    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.245619 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245094    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e6d96c6b-2765-4f69-934b-00641848c010-iptables-alerter-script\") pod \"iptables-alerter-crd6s\" (UID: \"e6d96c6b-2765-4f69-934b-00641848c010\") " pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245132    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-slash\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245156    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-etc-openvswitch\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245181    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-run-openvswitch\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245203    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-conf-dir\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245226    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9315e6c2-c11e-4ee7-a5fd-5f57e12772a6-host\") pod \"node-ca-qfqhg\" (UID: \"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6\") " pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245255    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-58dsl\" (UniqueName: \"kubernetes.io/projected/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-kube-api-access-58dsl\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245281    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-run-systemd\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245304    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245306    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c9057d1-fd80-4b1c-af89-acfff9aec573-ovn-node-metrics-cert\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245366    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-cni-netd\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245413    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-var-lib-kubelet\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245451    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-run-ovn-kubernetes\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245625    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e6d96c6b-2765-4f69-934b-00641848c010-iptables-alerter-script\") pod \"iptables-alerter-crd6s\" (UID: \"e6d96c6b-2765-4f69-934b-00641848c010\") " pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245651    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-hostroot\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245623    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245679    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-cni-dir\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245704    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-os-release\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.246285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245762    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-kubernetes\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245784    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/606fba4b-6867-42fe-b598-0dc46cf45c5b-agent-certs\") pod \"konnectivity-agent-45gzz\" (UID: \"606fba4b-6867-42fe-b598-0dc46cf45c5b\") " pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245806    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a052ebe-16e4-40a8-b822-8160f7735c0b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245833    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-sys-fs\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245859    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wknvp\" (UniqueName: \"kubernetes.io/projected/e6d96c6b-2765-4f69-934b-00641848c010-kube-api-access-wknvp\") pod \"iptables-alerter-crd6s\" (UID: \"e6d96c6b-2765-4f69-934b-00641848c010\") " pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245884    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c9057d1-fd80-4b1c-af89-acfff9aec573-env-overrides\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245903    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c9057d1-fd80-4b1c-af89-acfff9aec573-ovnkube-script-lib\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245908    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-cnibin\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245955    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-df6wx\" (UniqueName: \"kubernetes.io/projected/9315e6c2-c11e-4ee7-a5fd-5f57e12772a6-kube-api-access-df6wx\") pod \"node-ca-qfqhg\" (UID: \"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6\") " pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245977    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-node-log\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246006    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-lib-modules\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246005    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-slash\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246060    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-var-lib-cni-bin\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.246068    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs podName:c0d3ce5a-3b25-4284-8946-bbb3e5820ce6 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:33.746012855 +0000 UTC m=+3.073678951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs") pod "network-metrics-daemon-2dnl7" (UID: "c0d3ce5a-3b25-4284-8946-bbb3e5820ce6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246106    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-conf-dir\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246106    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-etc-kubernetes\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246130    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-run-openvswitch\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.247169 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246157    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9315e6c2-c11e-4ee7-a5fd-5f57e12772a6-host\") pod \"node-ca-qfqhg\" (UID: \"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6\") " pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246271    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-run-multus-certs\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246322    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-sysconfig\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246356    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-run-systemd\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246390    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-hostroot\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246437    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-sysctl-conf\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246443    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-cni-dir\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246490    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-device-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246502    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-os-release\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246574    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-socket-dir-parent\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246596    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-host-run-netns\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246614    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-run-k8s-cni-cncf-io\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.246652    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-run-netns\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247049    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9315e6c2-c11e-4ee7-a5fd-5f57e12772a6-serviceca\") pod \"node-ca-qfqhg\" (UID: \"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6\") " pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247087    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a052ebe-16e4-40a8-b822-8160f7735c0b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247167    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247225    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-registration-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247213    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-sys-fs\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.247938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247296    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-var-lib-cni-multus\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.245957    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-cnibin\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247445    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-node-log\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247565    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-etc-openvswitch\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247576    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-lib-modules\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247622    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-kubernetes\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247703    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-cnibin\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247738    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a052ebe-16e4-40a8-b822-8160f7735c0b-cni-binary-copy\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247764    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-var-lib-openvswitch\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247788    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-run\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247812    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-sys\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247837    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-var-lib-kubelet\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247849    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a8254b7-7a5c-4f77-9bf1-64605de1f147-cni-binary-copy\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247865    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbde4ba0-8948-4141-93a0-c0ff1aa62641-hosts-file\") pod \"node-resolver-7qxzh\" (UID: \"bbde4ba0-8948-4141-93a0-c0ff1aa62641\") " pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247891    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ttpp4\" (UniqueName: \"kubernetes.io/projected/bbde4ba0-8948-4141-93a0-c0ff1aa62641-kube-api-access-ttpp4\") pod \"node-resolver-7qxzh\" (UID: \"bbde4ba0-8948-4141-93a0-c0ff1aa62641\") " pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247917    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-os-release\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247927    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c9057d1-fd80-4b1c-af89-acfff9aec573-env-overrides\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247940    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-system-cni-dir\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.248719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247964    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbde4ba0-8948-4141-93a0-c0ff1aa62641-tmp-dir\") pod \"node-resolver-7qxzh\" (UID: \"bbde4ba0-8948-4141-93a0-c0ff1aa62641\") " pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247988    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a052ebe-16e4-40a8-b822-8160f7735c0b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.247995    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6d96c6b-2765-4f69-934b-00641848c010-host-slash\") pod \"iptables-alerter-crd6s\" (UID: \"e6d96c6b-2765-4f69-934b-00641848c010\") " pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248014    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-socket-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248084    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-etc-selinux\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248136    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-socket-dir\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248167    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-modprobe-d\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248177    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-cnibin\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248209    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-log-socket\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248260    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-etc-systemd\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.249436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248302    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-systemd-units\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.251574 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248646    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a052ebe-16e4-40a8-b822-8160f7735c0b-cni-binary-copy\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.251665 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248686    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9057d1-fd80-4b1c-af89-acfff9aec573-var-lib-openvswitch\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.251665 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248723    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-run\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.251665 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248757    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/777a4765-6b60-4801-95b7-f632242323c4-sys\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.251665 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248789    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-host-var-lib-kubelet\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.251665 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248788    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a8254b7-7a5c-4f77-9bf1-64605de1f147-multus-daemon-config\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.251665 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248825    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbde4ba0-8948-4141-93a0-c0ff1aa62641-hosts-file\") pod \"node-resolver-7qxzh\" (UID: \"bbde4ba0-8948-4141-93a0-c0ff1aa62641\") " pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.251665 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.248852    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8254b7-7a5c-4f77-9bf1-64605de1f147-system-cni-dir\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.251665 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.249081    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbde4ba0-8948-4141-93a0-c0ff1aa62641-tmp-dir\") pod \"node-resolver-7qxzh\" (UID: \"bbde4ba0-8948-4141-93a0-c0ff1aa62641\") " pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.251927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.249612    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/606fba4b-6867-42fe-b598-0dc46cf45c5b-agent-certs\") pod \"konnectivity-agent-45gzz\" (UID: \"606fba4b-6867-42fe-b598-0dc46cf45c5b\") " pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:33.251927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.249675    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a052ebe-16e4-40a8-b822-8160f7735c0b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.251927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.249697    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0a052ebe-16e4-40a8-b822-8160f7735c0b-os-release\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.251927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.249896    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c9057d1-fd80-4b1c-af89-acfff9aec573-ovn-node-metrics-cert\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.251927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.250267    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/777a4765-6b60-4801-95b7-f632242323c4-etc-tuned\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.251927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.251205    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/777a4765-6b60-4801-95b7-f632242323c4-tmp\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.255825 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.255804    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:33.255950 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.255828    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:33.255950 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.255843    2560 projected.go:194] Error preparing data for projected volume kube-api-access-ft8bw for pod openshift-network-diagnostics/network-check-target-4sgrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:33.255950 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.255932    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw podName:5dc8d16d-e5f7-410b-b433-0e30e23fd836 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:33.75590429 +0000 UTC m=+3.083570390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ft8bw" (UniqueName: "kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw") pod "network-check-target-4sgrj" (UID: "5dc8d16d-e5f7-410b-b433-0e30e23fd836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:33.256377 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.256353    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hv6\" (UniqueName: \"kubernetes.io/projected/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-kube-api-access-r6hv6\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:33.258433 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.258391    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bllx\" (UniqueName: \"kubernetes.io/projected/777a4765-6b60-4801-95b7-f632242323c4-kube-api-access-2bllx\") pod \"tuned-f5hsm\" (UID: \"777a4765-6b60-4801-95b7-f632242323c4\") " pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.258534 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.258462    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnk5t\" (UniqueName: \"kubernetes.io/projected/3c9057d1-fd80-4b1c-af89-acfff9aec573-kube-api-access-nnk5t\") pod \"ovnkube-node-t4zhg\" (UID: \"3c9057d1-fd80-4b1c-af89-acfff9aec573\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.259659 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.259629    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp2qs\" (UniqueName: \"kubernetes.io/projected/0a052ebe-16e4-40a8-b822-8160f7735c0b-kube-api-access-vp2qs\") pod \"multus-additional-cni-plugins-tk77g\" (UID: \"0a052ebe-16e4-40a8-b822-8160f7735c0b\") " pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.259659 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.260057    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58dsl\" (UniqueName: \"kubernetes.io/projected/b4db5df0-40cc-4b4b-b339-ebf4c88e840a-kube-api-access-58dsl\") pod \"aws-ebs-csi-driver-node-d49tg\" (UID: \"b4db5df0-40cc-4b4b-b339-ebf4c88e840a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.259659 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.260211    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6wx\" (UniqueName: \"kubernetes.io/projected/9315e6c2-c11e-4ee7-a5fd-5f57e12772a6-kube-api-access-df6wx\") pod \"node-ca-qfqhg\" (UID: \"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6\") " pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.260921 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.260709    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psd47\" (UniqueName: \"kubernetes.io/projected/1a8254b7-7a5c-4f77-9bf1-64605de1f147-kube-api-access-psd47\") pod \"multus-59tbq\" (UID: \"1a8254b7-7a5c-4f77-9bf1-64605de1f147\") " pod="openshift-multus/multus-59tbq"
May 28 00:11:33.261246 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.261222    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknvp\" (UniqueName: \"kubernetes.io/projected/e6d96c6b-2765-4f69-934b-00641848c010-kube-api-access-wknvp\") pod \"iptables-alerter-crd6s\" (UID: \"e6d96c6b-2765-4f69-934b-00641848c010\") " pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.265036 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.264959    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttpp4\" (UniqueName: \"kubernetes.io/projected/bbde4ba0-8948-4141-93a0-c0ff1aa62641-kube-api-access-ttpp4\") pod \"node-resolver-7qxzh\" (UID: \"bbde4ba0-8948-4141-93a0-c0ff1aa62641\") " pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.429097 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.429013    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-59tbq"
May 28 00:11:33.437958 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.437935    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f5hsm"
May 28 00:11:33.444458 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.444424    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qfqhg"
May 28 00:11:33.450927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.450905    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tk77g"
May 28 00:11:33.457470 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.457449    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg"
May 28 00:11:33.462984 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.462960    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7qxzh"
May 28 00:11:33.470482 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.470467    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crd6s"
May 28 00:11:33.476078 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.476061    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:33.481693 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.481674    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:33.490285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.490266    2560 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:33.752024 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.751995    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:33.752151 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.752131    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:33.752281 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.752207    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs podName:c0d3ce5a-3b25-4284-8946-bbb3e5820ce6 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:34.752187812 +0000 UTC m=+4.079853903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs") pod "network-metrics-daemon-2dnl7" (UID: "c0d3ce5a-3b25-4284-8946-bbb3e5820ce6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:33.764263 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:33.764239    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777a4765_6b60_4801_95b7_f632242323c4.slice/crio-0450fbd93248e437266d234a5dd01d702214c570a5203efe333ee2b3719a67ac WatchSource:0}: Error finding container 0450fbd93248e437266d234a5dd01d702214c570a5203efe333ee2b3719a67ac: Status 404 returned error can't find the container with id 0450fbd93248e437266d234a5dd01d702214c570a5203efe333ee2b3719a67ac
May 28 00:11:33.765277 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:33.765255    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4db5df0_40cc_4b4b_b339_ebf4c88e840a.slice/crio-943de57de2beeb6dd300fbbca8e29b97368c6602f97f24c9a985cc8bb5169afe WatchSource:0}: Error finding container 943de57de2beeb6dd300fbbca8e29b97368c6602f97f24c9a985cc8bb5169afe: Status 404 returned error can't find the container with id 943de57de2beeb6dd300fbbca8e29b97368c6602f97f24c9a985cc8bb5169afe
May 28 00:11:33.766239 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:33.766217    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606fba4b_6867_42fe_b598_0dc46cf45c5b.slice/crio-0c5ee67bfbc2ca9bc91ef5283054a79d8b5673282f48057cc179ab96d51272a3 WatchSource:0}: Error finding container 0c5ee67bfbc2ca9bc91ef5283054a79d8b5673282f48057cc179ab96d51272a3: Status 404 returned error can't find the container with id 0c5ee67bfbc2ca9bc91ef5283054a79d8b5673282f48057cc179ab96d51272a3
May 28 00:11:33.768095 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:33.768072    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d96c6b_2765_4f69_934b_00641848c010.slice/crio-faae26cd47af36af91e00ee29875e0b10b5a0e29888ad0a88d6a7b2c449f6a85 WatchSource:0}: Error finding container faae26cd47af36af91e00ee29875e0b10b5a0e29888ad0a88d6a7b2c449f6a85: Status 404 returned error can't find the container with id faae26cd47af36af91e00ee29875e0b10b5a0e29888ad0a88d6a7b2c449f6a85
May 28 00:11:33.768581 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:33.768561    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a052ebe_16e4_40a8_b822_8160f7735c0b.slice/crio-91dcf3b952faff69f66d548b671c43983a4259e6325b69541ca363fb2aebb8a3 WatchSource:0}: Error finding container 91dcf3b952faff69f66d548b671c43983a4259e6325b69541ca363fb2aebb8a3: Status 404 returned error can't find the container with id 91dcf3b952faff69f66d548b671c43983a4259e6325b69541ca363fb2aebb8a3
May 28 00:11:33.771493 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:33.771474    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8254b7_7a5c_4f77_9bf1_64605de1f147.slice/crio-2380e29bddc2df2a3460508d4d5fa65e43a283e686a89d8ce39cc6726477a64b WatchSource:0}: Error finding container 2380e29bddc2df2a3460508d4d5fa65e43a283e686a89d8ce39cc6726477a64b: Status 404 returned error can't find the container with id 2380e29bddc2df2a3460508d4d5fa65e43a283e686a89d8ce39cc6726477a64b
May 28 00:11:33.772479 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:33.772443    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbde4ba0_8948_4141_93a0_c0ff1aa62641.slice/crio-d20748043f1d6b7e30b0f414adc9ba481996a56ce0c7b8a52ea4c8516d334a73 WatchSource:0}: Error finding container d20748043f1d6b7e30b0f414adc9ba481996a56ce0c7b8a52ea4c8516d334a73: Status 404 returned error can't find the container with id d20748043f1d6b7e30b0f414adc9ba481996a56ce0c7b8a52ea4c8516d334a73
May 28 00:11:33.773142 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:33.773122    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9057d1_fd80_4b1c_af89_acfff9aec573.slice/crio-d116475fe4b1e5cc8c175366f8a7a874739a951bcb6095e428d1a4792c25771d WatchSource:0}: Error finding container d116475fe4b1e5cc8c175366f8a7a874739a951bcb6095e428d1a4792c25771d: Status 404 returned error can't find the container with id d116475fe4b1e5cc8c175366f8a7a874739a951bcb6095e428d1a4792c25771d
May 28 00:11:33.775971 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:11:33.775866    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9315e6c2_c11e_4ee7_a5fd_5f57e12772a6.slice/crio-15ded3acf20df4bc5dbaafd5f2c4a4075b0502bca5a3110c76cc509e36d7fbdf WatchSource:0}: Error finding container 15ded3acf20df4bc5dbaafd5f2c4a4075b0502bca5a3110c76cc509e36d7fbdf: Status 404 returned error can't find the container with id 15ded3acf20df4bc5dbaafd5f2c4a4075b0502bca5a3110c76cc509e36d7fbdf
May 28 00:11:33.852409 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:33.852296    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:33.852553 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.852442    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:33.852553 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.852459    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:33.852553 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.852469    2560 projected.go:194] Error preparing data for projected volume kube-api-access-ft8bw for pod openshift-network-diagnostics/network-check-target-4sgrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:33.852553 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:33.852545    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw podName:5dc8d16d-e5f7-410b-b433-0e30e23fd836 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:34.852506277 +0000 UTC m=+4.180172366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft8bw" (UniqueName: "kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw") pod "network-check-target-4sgrj" (UID: "5dc8d16d-e5f7-410b-b433-0e30e23fd836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:34.174854 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.174812    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 00:06:32 +0000 UTC" deadline="2028-01-02 11:11:17.914906561 +0000 UTC"
May 28 00:11:34.174854 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.174852    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14026h59m43.740058329s"
May 28 00:11:34.283164 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.283089    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" event={"ID":"3c9057d1-fd80-4b1c-af89-acfff9aec573","Type":"ContainerStarted","Data":"d116475fe4b1e5cc8c175366f8a7a874739a951bcb6095e428d1a4792c25771d"}
May 28 00:11:34.287013 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.286959    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-59tbq" event={"ID":"1a8254b7-7a5c-4f77-9bf1-64605de1f147","Type":"ContainerStarted","Data":"2380e29bddc2df2a3460508d4d5fa65e43a283e686a89d8ce39cc6726477a64b"}
May 28 00:11:34.292277 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.292214    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crd6s" event={"ID":"e6d96c6b-2765-4f69-934b-00641848c010","Type":"ContainerStarted","Data":"faae26cd47af36af91e00ee29875e0b10b5a0e29888ad0a88d6a7b2c449f6a85"}
May 28 00:11:34.300688 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.300510    2560 generic.go:358] "Generic (PLEG): container finished" podID="0471fde4bca3037bb39fa531a41ff217" containerID="28253582150b7a873370a264587ba151f0341d1407868334a30d343b3a5a7ed3" exitCode=0
May 28 00:11:34.300688 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.300654    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal" event={"ID":"0471fde4bca3037bb39fa531a41ff217","Type":"ContainerDied","Data":"28253582150b7a873370a264587ba151f0341d1407868334a30d343b3a5a7ed3"}
May 28 00:11:34.314306 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.314039    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7qxzh" event={"ID":"bbde4ba0-8948-4141-93a0-c0ff1aa62641","Type":"ContainerStarted","Data":"d20748043f1d6b7e30b0f414adc9ba481996a56ce0c7b8a52ea4c8516d334a73"}
May 28 00:11:34.320437 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.320383    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tk77g" event={"ID":"0a052ebe-16e4-40a8-b822-8160f7735c0b","Type":"ContainerStarted","Data":"91dcf3b952faff69f66d548b671c43983a4259e6325b69541ca363fb2aebb8a3"}
May 28 00:11:34.335879 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.335829    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-45gzz" event={"ID":"606fba4b-6867-42fe-b598-0dc46cf45c5b","Type":"ContainerStarted","Data":"0c5ee67bfbc2ca9bc91ef5283054a79d8b5673282f48057cc179ab96d51272a3"}
May 28 00:11:34.352746 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.352714    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg" event={"ID":"b4db5df0-40cc-4b4b-b339-ebf4c88e840a","Type":"ContainerStarted","Data":"943de57de2beeb6dd300fbbca8e29b97368c6602f97f24c9a985cc8bb5169afe"}
May 28 00:11:34.370690 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.370661    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f5hsm" event={"ID":"777a4765-6b60-4801-95b7-f632242323c4","Type":"ContainerStarted","Data":"0450fbd93248e437266d234a5dd01d702214c570a5203efe333ee2b3719a67ac"}
May 28 00:11:34.388681 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.388650    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal" event={"ID":"6fac2f13116ec505f7f49d6272df8583","Type":"ContainerStarted","Data":"8ffb2a7a3e15a3d53281494c5f564158d942f5f8fc58586208e8148a8f232777"}
May 28 00:11:34.407231 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.407194    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qfqhg" event={"ID":"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6","Type":"ContainerStarted","Data":"15ded3acf20df4bc5dbaafd5f2c4a4075b0502bca5a3110c76cc509e36d7fbdf"}
May 28 00:11:34.759417 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.759371    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:34.759614 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:34.759562    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:34.759688 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:34.759626    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs podName:c0d3ce5a-3b25-4284-8946-bbb3e5820ce6 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:36.759607767 +0000 UTC m=+6.087273859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs") pod "network-metrics-daemon-2dnl7" (UID: "c0d3ce5a-3b25-4284-8946-bbb3e5820ce6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:34.860649 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:34.859981    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:34.860649 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:34.860146    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:34.860649 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:34.860168    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:34.860649 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:34.860180    2560 projected.go:194] Error preparing data for projected volume kube-api-access-ft8bw for pod openshift-network-diagnostics/network-check-target-4sgrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:34.860649 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:34.860241    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw podName:5dc8d16d-e5f7-410b-b433-0e30e23fd836 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:36.860221676 +0000 UTC m=+6.187887764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft8bw" (UniqueName: "kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw") pod "network-check-target-4sgrj" (UID: "5dc8d16d-e5f7-410b-b433-0e30e23fd836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:35.258377 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:35.257685    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:35.258377 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:35.257905    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:35.259119 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:35.258980    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:35.259119 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:35.259078    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:35.426220 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:35.426178    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal" event={"ID":"0471fde4bca3037bb39fa531a41ff217","Type":"ContainerStarted","Data":"837049769a5e731b1afebe277e1bd104287a1520d5915a2ca9e753ebe57ee4ed"}
May 28 00:11:35.440977 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:35.440917    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal" podStartSLOduration=3.440900508 podStartE2EDuration="3.440900508s" podCreationTimestamp="2026-05-28 00:11:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:11:34.402588475 +0000 UTC m=+3.730254583" watchObservedRunningTime="2026-05-28 00:11:35.440900508 +0000 UTC m=+4.768566616"
May 28 00:11:36.777661 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:36.777562    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:36.778083 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:36.777742    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:36.778083 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:36.777814    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs podName:c0d3ce5a-3b25-4284-8946-bbb3e5820ce6 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:40.777798653 +0000 UTC m=+10.105464739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs") pod "network-metrics-daemon-2dnl7" (UID: "c0d3ce5a-3b25-4284-8946-bbb3e5820ce6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:36.878356 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:36.878316    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:36.878538 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:36.878503    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:36.878642 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:36.878545    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:36.878642 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:36.878559    2560 projected.go:194] Error preparing data for projected volume kube-api-access-ft8bw for pod openshift-network-diagnostics/network-check-target-4sgrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:36.878642 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:36.878618    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw podName:5dc8d16d-e5f7-410b-b433-0e30e23fd836 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:40.878599943 +0000 UTC m=+10.206266047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft8bw" (UniqueName: "kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw") pod "network-check-target-4sgrj" (UID: "5dc8d16d-e5f7-410b-b433-0e30e23fd836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:37.259193 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:37.259160    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:37.259350 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:37.259259    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:37.259703 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:37.259681    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:37.259842 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:37.259780    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:39.256891 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:39.256361    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:39.256891 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:39.256493    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:39.257350 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:39.257000    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:39.257350 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:39.257117    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:40.812730 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:40.812655    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:40.813163 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:40.812822    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:40.813163 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:40.812895    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs podName:c0d3ce5a-3b25-4284-8946-bbb3e5820ce6 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:48.812873934 +0000 UTC m=+18.140540020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs") pod "network-metrics-daemon-2dnl7" (UID: "c0d3ce5a-3b25-4284-8946-bbb3e5820ce6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:40.914438 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:40.913865    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:40.914438 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:40.914027    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:40.914438 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:40.914042    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:40.914438 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:40.914051    2560 projected.go:194] Error preparing data for projected volume kube-api-access-ft8bw for pod openshift-network-diagnostics/network-check-target-4sgrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:40.914438 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:40.914100    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw podName:5dc8d16d-e5f7-410b-b433-0e30e23fd836 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:48.914081304 +0000 UTC m=+18.241747403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft8bw" (UniqueName: "kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw") pod "network-check-target-4sgrj" (UID: "5dc8d16d-e5f7-410b-b433-0e30e23fd836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:41.259015 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:41.258040    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:41.259015 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:41.258144    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:41.259015 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:41.258234    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:41.259015 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:41.258342    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:43.256966 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:43.256922    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:43.257314 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:43.256942    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:43.257314 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:43.257046    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:43.257314 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:43.257117    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:45.256922 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:45.256886    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:45.257410 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:45.256886    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:45.257410 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:45.257008    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:45.257410 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:45.257080    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:47.256326 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:47.256285    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:47.256746 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:47.256286    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:47.256746 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:47.256413    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:47.256746 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:47.256507    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:48.876353 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:48.876318    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:48.876913 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:48.876504    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:48.876913 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:48.876609    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs podName:c0d3ce5a-3b25-4284-8946-bbb3e5820ce6 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:04.876585797 +0000 UTC m=+34.204251902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs") pod "network-metrics-daemon-2dnl7" (UID: "c0d3ce5a-3b25-4284-8946-bbb3e5820ce6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:48.976954 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:48.976920    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:48.977123 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:48.977094    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:48.977123 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:48.977120    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:48.977212 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:48.977133    2560 projected.go:194] Error preparing data for projected volume kube-api-access-ft8bw for pod openshift-network-diagnostics/network-check-target-4sgrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:48.977212 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:48.977190    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw podName:5dc8d16d-e5f7-410b-b433-0e30e23fd836 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:04.977171565 +0000 UTC m=+34.304837657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft8bw" (UniqueName: "kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw") pod "network-check-target-4sgrj" (UID: "5dc8d16d-e5f7-410b-b433-0e30e23fd836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:49.256787 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:49.256699    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:49.256942 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:49.256713    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:49.256942 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:49.256848    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:49.256942 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:49.256912    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:51.256920 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:51.256881    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:51.257360 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:51.256989    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:51.257360 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:51.257075    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:51.257360 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:51.257202    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:52.461140 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:52.461107    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7qxzh" event={"ID":"bbde4ba0-8948-4141-93a0-c0ff1aa62641","Type":"ContainerStarted","Data":"794e87eb4f187ec3ceff2552d4cbe5c900a0d2bce324002a4936291327dcf33e"}
May 28 00:11:52.467111 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:52.466808    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-45gzz" event={"ID":"606fba4b-6867-42fe-b598-0dc46cf45c5b","Type":"ContainerStarted","Data":"f8b59e44ca7ae6d2802ab235cffca39ec4502bdba37f1c43d8f812cf5bcdae12"}
May 28 00:11:52.474747 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:52.474409    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qfqhg" event={"ID":"9315e6c2-c11e-4ee7-a5fd-5f57e12772a6","Type":"ContainerStarted","Data":"a0e0f06350f43cbe96870912a84aa0c9399a9f746ead887d62ca5cffdc963d6b"}
May 28 00:11:52.477170 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:52.476962    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-7qxzh" podStartSLOduration=3.2395234569999998 podStartE2EDuration="21.47692898s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:11:33.775096256 +0000 UTC m=+3.102762343" lastFinishedPulling="2026-05-28 00:11:52.012501762 +0000 UTC m=+21.340167866" observedRunningTime="2026-05-28 00:11:52.476863596 +0000 UTC m=+21.804529704" watchObservedRunningTime="2026-05-28 00:11:52.47692898 +0000 UTC m=+21.804595068"
May 28 00:11:52.477832 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:52.477677    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal" podStartSLOduration=20.477660201 podStartE2EDuration="20.477660201s" podCreationTimestamp="2026-05-28 00:11:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:11:35.441319786 +0000 UTC m=+4.768985895" watchObservedRunningTime="2026-05-28 00:11:52.477660201 +0000 UTC m=+21.805326310"
May 28 00:11:52.504854 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:52.504762    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qfqhg" podStartSLOduration=3.208908416 podStartE2EDuration="21.504744629s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:11:33.779202922 +0000 UTC m=+3.106869008" lastFinishedPulling="2026-05-28 00:11:52.075039128 +0000 UTC m=+21.402705221" observedRunningTime="2026-05-28 00:11:52.492175871 +0000 UTC m=+21.819841980" watchObservedRunningTime="2026-05-28 00:11:52.504744629 +0000 UTC m=+21.832410738"
May 28 00:11:53.257339 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.257053    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:53.257491 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.257128    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:53.257491 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:53.257397    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:53.257491 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:53.257464    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:53.356039 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.356012    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:53.477588 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.477565    2560 generic.go:358] "Generic (PLEG): container finished" podID="0a052ebe-16e4-40a8-b822-8160f7735c0b" containerID="3e499d8291c4f87709aa841d79b352c9d0613b6e793bdd5dd591b4bd2dac0636" exitCode=0
May 28 00:11:53.478176 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.477624    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tk77g" event={"ID":"0a052ebe-16e4-40a8-b822-8160f7735c0b","Type":"ContainerDied","Data":"3e499d8291c4f87709aa841d79b352c9d0613b6e793bdd5dd591b4bd2dac0636"}
May 28 00:11:53.478878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.478847    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg" event={"ID":"b4db5df0-40cc-4b4b-b339-ebf4c88e840a","Type":"ContainerStarted","Data":"14a8ad6c20f4c71e02611d636a6908782cb7dde041661142d8bd73305368b462"}
May 28 00:11:53.480214 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.480175    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f5hsm" event={"ID":"777a4765-6b60-4801-95b7-f632242323c4","Type":"ContainerStarted","Data":"212f6a15e290f48e43ee99d0ab9cccebaec89057acc32d8abef3d209f133b204"}
May 28 00:11:53.485977 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.485957    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" event={"ID":"3c9057d1-fd80-4b1c-af89-acfff9aec573","Type":"ContainerStarted","Data":"449ff5cb8253127201a38fbaea03fbf6bbe3d29e2bd7a2fac7a84f7e8d731e76"}
May 28 00:11:53.486056 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.485980    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" event={"ID":"3c9057d1-fd80-4b1c-af89-acfff9aec573","Type":"ContainerStarted","Data":"38ead6656b36eaed0ee823d8d47beb4b72b4ca688ef29e23bb4f8314d55fc070"}
May 28 00:11:53.486056 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.485990    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" event={"ID":"3c9057d1-fd80-4b1c-af89-acfff9aec573","Type":"ContainerStarted","Data":"fbd35135150e6a9fc0aae829419ac1139534cce5c880517d99c4c6e4d254fdc8"}
May 28 00:11:53.486056 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.485998    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" event={"ID":"3c9057d1-fd80-4b1c-af89-acfff9aec573","Type":"ContainerStarted","Data":"ff67c1dda3c89a27d6ad0fc899e02488407762941a09ec0989cb35f706a5b88d"}
May 28 00:11:53.486056 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.486005    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" event={"ID":"3c9057d1-fd80-4b1c-af89-acfff9aec573","Type":"ContainerStarted","Data":"b739bbb18d4d0b51bf7a3edb5e3fc8934b83c33fe8bcff92f23713aab64893e1"}
May 28 00:11:53.486056 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.486012    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" event={"ID":"3c9057d1-fd80-4b1c-af89-acfff9aec573","Type":"ContainerStarted","Data":"18ad030dd0fbec281f531b8ded60fbd0552b205d0e745e572886d30df06ebb78"}
May 28 00:11:53.486990 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.486971    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-59tbq" event={"ID":"1a8254b7-7a5c-4f77-9bf1-64605de1f147","Type":"ContainerStarted","Data":"0c6f19974bc4c9a18fa64a3ee5cad9f21117a7ccd6b979c12a40db19fa304011"}
May 28 00:11:53.488269 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.488242    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crd6s" event={"ID":"e6d96c6b-2765-4f69-934b-00641848c010","Type":"ContainerStarted","Data":"595dbd7196f033aabb720104bc1037114283bef1ea1bc50d3aa2ba63452ee6dd"}
May 28 00:11:53.500730 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.500695    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-45gzz" podStartSLOduration=4.193553123 podStartE2EDuration="22.500685035s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:11:33.767900017 +0000 UTC m=+3.095566102" lastFinishedPulling="2026-05-28 00:11:52.075031924 +0000 UTC m=+21.402698014" observedRunningTime="2026-05-28 00:11:52.504839461 +0000 UTC m=+21.832505576" watchObservedRunningTime="2026-05-28 00:11:53.500685035 +0000 UTC m=+22.828351143"
May 28 00:11:53.508476 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.508433    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 28 00:11:53.528264 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.528221    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f5hsm" podStartSLOduration=4.103569053 podStartE2EDuration="22.528211069s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:11:33.766431218 +0000 UTC m=+3.094097307" lastFinishedPulling="2026-05-28 00:11:52.191073222 +0000 UTC m=+21.518739323" observedRunningTime="2026-05-28 00:11:53.514584079 +0000 UTC m=+22.842250189" watchObservedRunningTime="2026-05-28 00:11:53.528211069 +0000 UTC m=+22.855877179"
May 28 00:11:53.528693 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.528668    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-crd6s" podStartSLOduration=4.109176719 podStartE2EDuration="22.528663503s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:11:33.769849533 +0000 UTC m=+3.097515622" lastFinishedPulling="2026-05-28 00:11:52.189336306 +0000 UTC m=+21.517002406" observedRunningTime="2026-05-28 00:11:53.528324882 +0000 UTC m=+22.855990989" watchObservedRunningTime="2026-05-28 00:11:53.528663503 +0000 UTC m=+22.856329611"
May 28 00:11:53.543092 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:53.543060    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-59tbq" podStartSLOduration=4.087172833 podStartE2EDuration="22.543054449s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:11:33.773369302 +0000 UTC m=+3.101035388" lastFinishedPulling="2026-05-28 00:11:52.229250913 +0000 UTC m=+21.556917004" observedRunningTime="2026-05-28 00:11:53.542623777 +0000 UTC m=+22.870289885" watchObservedRunningTime="2026-05-28 00:11:53.543054449 +0000 UTC m=+22.870720554"
May 28 00:11:54.216584 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:54.216441    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-28T00:11:53.50844654Z","UUID":"9b9e1015-3338-4bd6-b726-d0e995163ea8","Handler":null,"Name":"","Endpoint":""}
May 28 00:11:54.218529 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:54.218489    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 28 00:11:54.218674 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:54.218541    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 28 00:11:54.492570 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:54.492469    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg" event={"ID":"b4db5df0-40cc-4b4b-b339-ebf4c88e840a","Type":"ContainerStarted","Data":"eb6c6c61bba3fc67e17d39902f809567f4b0a43d0d40bbce829a87d8be2d38bc"}
May 28 00:11:54.492570 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:54.492539    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg" event={"ID":"b4db5df0-40cc-4b4b-b339-ebf4c88e840a","Type":"ContainerStarted","Data":"304f590f1974273e3def21fc24c0ac571bbfa43617366112a8b4a7620b2e7039"}
May 28 00:11:55.178025 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:55.177948    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:55.178622 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:55.178588    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:55.197529 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:55.197468    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d49tg" podStartSLOduration=3.652462585 podStartE2EDuration="24.197456273s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:11:33.767354652 +0000 UTC m=+3.095020755" lastFinishedPulling="2026-05-28 00:11:54.312348342 +0000 UTC m=+23.640014443" observedRunningTime="2026-05-28 00:11:54.509701202 +0000 UTC m=+23.837367340" watchObservedRunningTime="2026-05-28 00:11:55.197456273 +0000 UTC m=+24.525122384"
May 28 00:11:55.256787 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:55.256755    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:55.256947 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:55.256895    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:55.256947 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:55.256937    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:55.257053 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:55.257023    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:55.497364 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:55.497065    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" event={"ID":"3c9057d1-fd80-4b1c-af89-acfff9aec573","Type":"ContainerStarted","Data":"e9bffc0865260e968aa6babdf1508c65e937d8e49a5f80a88a291b67759c5b95"}
May 28 00:11:55.497839 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:55.497822    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-45gzz"
May 28 00:11:57.256419 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:57.256347    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:57.257028 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:57.256451    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:57.257028 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:57.256477    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:57.257028 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:57.256543    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:57.502321 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:57.502104    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tk77g" event={"ID":"0a052ebe-16e4-40a8-b822-8160f7735c0b","Type":"ContainerStarted","Data":"781c4ce331801984246f8aeef91a2d8b4f0518881a04f131c59b4fb95993bdcd"}
May 28 00:11:57.505450 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:57.505423    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" event={"ID":"3c9057d1-fd80-4b1c-af89-acfff9aec573","Type":"ContainerStarted","Data":"fcebbb1efba9eb1a35e643bec1aa39cfdb79be5de85e908e48891d637da100a3"}
May 28 00:11:57.505739 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:57.505702    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:57.505739 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:57.505734    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:57.521961 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:57.521820    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:57.552538 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:57.551826    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg" podStartSLOduration=8.106558352 podStartE2EDuration="26.551809533s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:11:33.776880695 +0000 UTC m=+3.104546791" lastFinishedPulling="2026-05-28 00:11:52.222131873 +0000 UTC m=+21.549797972" observedRunningTime="2026-05-28 00:11:57.550574276 +0000 UTC m=+26.878240385" watchObservedRunningTime="2026-05-28 00:11:57.551809533 +0000 UTC m=+26.879475643"
May 28 00:11:58.509734 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:58.509701    2560 generic.go:358] "Generic (PLEG): container finished" podID="0a052ebe-16e4-40a8-b822-8160f7735c0b" containerID="781c4ce331801984246f8aeef91a2d8b4f0518881a04f131c59b4fb95993bdcd" exitCode=0
May 28 00:11:58.510420 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:58.509768    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tk77g" event={"ID":"0a052ebe-16e4-40a8-b822-8160f7735c0b","Type":"ContainerDied","Data":"781c4ce331801984246f8aeef91a2d8b4f0518881a04f131c59b4fb95993bdcd"}
May 28 00:11:58.510580 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:58.510509    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:58.527769 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:58.527747    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:11:59.256944 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:59.256916    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:59.257082 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:59.257055    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:59.257425 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:59.257402    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:59.257507 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:59.257489    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:11:59.478640 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:59.478610    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4sgrj"]
May 28 00:11:59.481593 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:59.481569    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2dnl7"]
May 28 00:11:59.513445 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:59.513383    2560 generic.go:358] "Generic (PLEG): container finished" podID="0a052ebe-16e4-40a8-b822-8160f7735c0b" containerID="cf15f33a02d21165a989b13976dba6e2963f934bb5e695507aa36680cf4ec77b" exitCode=0
May 28 00:11:59.513809 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:59.513483    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tk77g" event={"ID":"0a052ebe-16e4-40a8-b822-8160f7735c0b","Type":"ContainerDied","Data":"cf15f33a02d21165a989b13976dba6e2963f934bb5e695507aa36680cf4ec77b"}
May 28 00:11:59.513809 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:59.513562    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:11:59.513809 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:59.513669    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:11:59.514700 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:11:59.514217    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:11:59.514700 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:11:59.514316    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:12:01.259616 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:01.259547    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:12:01.259988 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:01.259547    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:12:01.259988 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:01.259643    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:12:01.259988 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:01.259715    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:12:01.520298 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:01.520223    2560 generic.go:358] "Generic (PLEG): container finished" podID="0a052ebe-16e4-40a8-b822-8160f7735c0b" containerID="4f5cab1d0c298da579255c07ec8dbd3d89f0e35dc71cc3d616db07f7a3411a78" exitCode=0
May 28 00:12:01.520298 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:01.520280    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tk77g" event={"ID":"0a052ebe-16e4-40a8-b822-8160f7735c0b","Type":"ContainerDied","Data":"4f5cab1d0c298da579255c07ec8dbd3d89f0e35dc71cc3d616db07f7a3411a78"}
May 28 00:12:03.257178 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:03.257139    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:12:03.257178 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:03.257169    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:12:03.257925 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:03.257296    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sgrj" podUID="5dc8d16d-e5f7-410b-b433-0e30e23fd836"
May 28 00:12:03.257925 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:03.257401    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:12:04.896436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:04.896394    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:12:04.896940 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:04.896580    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:12:04.896940 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:04.896653    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs podName:c0d3ce5a-3b25-4284-8946-bbb3e5820ce6 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:36.896630731 +0000 UTC m=+66.224296825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs") pod "network-metrics-daemon-2dnl7" (UID: "c0d3ce5a-3b25-4284-8946-bbb3e5820ce6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:12:04.996964 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:04.996925    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:12:04.997137 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:04.997080    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:12:04.997137 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:04.997099    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:12:04.997137 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:04.997110    2560 projected.go:194] Error preparing data for projected volume kube-api-access-ft8bw for pod openshift-network-diagnostics/network-check-target-4sgrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:12:04.997280 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:04.997175    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw podName:5dc8d16d-e5f7-410b-b433-0e30e23fd836 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:36.997154964 +0000 UTC m=+66.324821069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft8bw" (UniqueName: "kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw") pod "network-check-target-4sgrj" (UID: "5dc8d16d-e5f7-410b-b433-0e30e23fd836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:12:05.018924 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.018901    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeReady"
May 28 00:12:05.019067 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.019043    2560 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 28 00:12:05.064277 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.064216    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nkppb"]
May 28 00:12:05.090307 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.090278    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4sppr"]
May 28 00:12:05.090443 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.090412    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.092825 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.092629    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 28 00:12:05.092825 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.092681    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 28 00:12:05.096156 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.093567    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzrg4\""
May 28 00:12:05.106299 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.106278    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nkppb"]
May 28 00:12:05.106417 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.106305    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4sppr"]
May 28 00:12:05.106417 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.106386    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:05.108442 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.108426    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 28 00:12:05.108625 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.108606    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 28 00:12:05.108721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.108703    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nx64p\""
May 28 00:12:05.108847 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.108831    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 28 00:12:05.198617 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.198583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:05.198617 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.198620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.198842 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.198650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200a652c-a7c2-4aee-8767-26801182f407-tmp-dir\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.198842 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.198752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/200a652c-a7c2-4aee-8767-26801182f407-config-volume\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.198842 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.198789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lt97\" (UniqueName: \"kubernetes.io/projected/200a652c-a7c2-4aee-8767-26801182f407-kube-api-access-2lt97\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.198960 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.198861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p529d\" (UniqueName: \"kubernetes.io/projected/da2bacc2-9142-49d3-a23c-a62ba15e855e-kube-api-access-p529d\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:05.256858 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.256818    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:12:05.257025 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.256816    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:12:05.259150 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.259127    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 00:12:05.259255 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.259154    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 00:12:05.259255 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.259194    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fp9tk\""
May 28 00:12:05.259464 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.259450    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 00:12:05.259508 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.259471    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q24q2\""
May 28 00:12:05.300194 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.300160    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p529d\" (UniqueName: \"kubernetes.io/projected/da2bacc2-9142-49d3-a23c-a62ba15e855e-kube-api-access-p529d\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:05.300369 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.300217    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:05.300369 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:05.300319    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:05.300478 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:05.300444    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert podName:da2bacc2-9142-49d3-a23c-a62ba15e855e nodeName:}" failed. No retries permitted until 2026-05-28 00:12:05.800421129 +0000 UTC m=+35.128087218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert") pod "ingress-canary-4sppr" (UID: "da2bacc2-9142-49d3-a23c-a62ba15e855e") : secret "canary-serving-cert" not found
May 28 00:12:05.300615 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.300595    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.300675 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.300650    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200a652c-a7c2-4aee-8767-26801182f407-tmp-dir\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.300724 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.300694    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/200a652c-a7c2-4aee-8767-26801182f407-config-volume\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.300783 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.300723    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2lt97\" (UniqueName: \"kubernetes.io/projected/200a652c-a7c2-4aee-8767-26801182f407-kube-api-access-2lt97\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.300834 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:05.300784    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:05.300882 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:05.300835    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls podName:200a652c-a7c2-4aee-8767-26801182f407 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:05.80081827 +0000 UTC m=+35.128484359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls") pod "dns-default-nkppb" (UID: "200a652c-a7c2-4aee-8767-26801182f407") : secret "dns-default-metrics-tls" not found
May 28 00:12:05.301153 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.301128    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200a652c-a7c2-4aee-8767-26801182f407-tmp-dir\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.301391 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.301374    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/200a652c-a7c2-4aee-8767-26801182f407-config-volume\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.310816 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.310794    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lt97\" (UniqueName: \"kubernetes.io/projected/200a652c-a7c2-4aee-8767-26801182f407-kube-api-access-2lt97\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.311018 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.310999    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p529d\" (UniqueName: \"kubernetes.io/projected/da2bacc2-9142-49d3-a23c-a62ba15e855e-kube-api-access-p529d\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:05.804717 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.804680    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:05.804894 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:05.804788    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:05.804894 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:05.804851    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:05.804894 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:05.804878    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:05.805049 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:05.804939    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls podName:200a652c-a7c2-4aee-8767-26801182f407 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:06.804917554 +0000 UTC m=+36.132583643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls") pod "dns-default-nkppb" (UID: "200a652c-a7c2-4aee-8767-26801182f407") : secret "dns-default-metrics-tls" not found
May 28 00:12:05.805049 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:05.804960    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert podName:da2bacc2-9142-49d3-a23c-a62ba15e855e nodeName:}" failed. No retries permitted until 2026-05-28 00:12:06.804950099 +0000 UTC m=+36.132616189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert") pod "ingress-canary-4sppr" (UID: "da2bacc2-9142-49d3-a23c-a62ba15e855e") : secret "canary-serving-cert" not found
May 28 00:12:06.811939 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:06.811904    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:06.812467 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:06.811950    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:06.812467 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:06.812075    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:06.812467 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:06.812079    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:06.812467 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:06.812148    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert podName:da2bacc2-9142-49d3-a23c-a62ba15e855e nodeName:}" failed. No retries permitted until 2026-05-28 00:12:08.812128539 +0000 UTC m=+38.139794627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert") pod "ingress-canary-4sppr" (UID: "da2bacc2-9142-49d3-a23c-a62ba15e855e") : secret "canary-serving-cert" not found
May 28 00:12:06.812467 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:06.812207    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls podName:200a652c-a7c2-4aee-8767-26801182f407 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:08.812185142 +0000 UTC m=+38.139851243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls") pod "dns-default-nkppb" (UID: "200a652c-a7c2-4aee-8767-26801182f407") : secret "dns-default-metrics-tls" not found
May 28 00:12:08.536218 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:08.536185    2560 generic.go:358] "Generic (PLEG): container finished" podID="0a052ebe-16e4-40a8-b822-8160f7735c0b" containerID="45d4bc50f943deb84c45762d347268100fda2e6c015f2a50a66cb2a791ce239d" exitCode=0
May 28 00:12:08.536592 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:08.536244    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tk77g" event={"ID":"0a052ebe-16e4-40a8-b822-8160f7735c0b","Type":"ContainerDied","Data":"45d4bc50f943deb84c45762d347268100fda2e6c015f2a50a66cb2a791ce239d"}
May 28 00:12:08.827174 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:08.827096    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:08.827174 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:08.827134    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:08.827334 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:08.827250    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:08.827334 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:08.827251    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:08.827334 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:08.827305    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls podName:200a652c-a7c2-4aee-8767-26801182f407 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:12.827289027 +0000 UTC m=+42.154955116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls") pod "dns-default-nkppb" (UID: "200a652c-a7c2-4aee-8767-26801182f407") : secret "dns-default-metrics-tls" not found
May 28 00:12:08.827334 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:08.827321    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert podName:da2bacc2-9142-49d3-a23c-a62ba15e855e nodeName:}" failed. No retries permitted until 2026-05-28 00:12:12.827314901 +0000 UTC m=+42.154980987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert") pod "ingress-canary-4sppr" (UID: "da2bacc2-9142-49d3-a23c-a62ba15e855e") : secret "canary-serving-cert" not found
May 28 00:12:09.541058 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:09.541030    2560 generic.go:358] "Generic (PLEG): container finished" podID="0a052ebe-16e4-40a8-b822-8160f7735c0b" containerID="38ceb7e1a26a04c3cf9a03855becf466091a800af875d312e99b8f8ea4f8bc5e" exitCode=0
May 28 00:12:09.541405 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:09.541073    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tk77g" event={"ID":"0a052ebe-16e4-40a8-b822-8160f7735c0b","Type":"ContainerDied","Data":"38ceb7e1a26a04c3cf9a03855becf466091a800af875d312e99b8f8ea4f8bc5e"}
May 28 00:12:10.545996 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:10.545966    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tk77g" event={"ID":"0a052ebe-16e4-40a8-b822-8160f7735c0b","Type":"ContainerStarted","Data":"161793f4b70618d8e280064f7fec6a8689c904502a1d0c481fb623cf841965b9"}
May 28 00:12:10.567396 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:10.567334    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tk77g" podStartSLOduration=5.7733456180000005 podStartE2EDuration="39.567320095s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:11:33.770361537 +0000 UTC m=+3.098027644" lastFinishedPulling="2026-05-28 00:12:07.564336036 +0000 UTC m=+36.892002121" observedRunningTime="2026-05-28 00:12:10.56674658 +0000 UTC m=+39.894412689" watchObservedRunningTime="2026-05-28 00:12:10.567320095 +0000 UTC m=+39.894986202"
May 28 00:12:12.854839 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:12.854780    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:12.854839 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:12.854835    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:12.855359 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:12.854933    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:12.855359 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:12.854938    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:12.855359 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:12.854986    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls podName:200a652c-a7c2-4aee-8767-26801182f407 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:20.854973321 +0000 UTC m=+50.182639407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls") pod "dns-default-nkppb" (UID: "200a652c-a7c2-4aee-8767-26801182f407") : secret "dns-default-metrics-tls" not found
May 28 00:12:12.855359 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:12.854999    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert podName:da2bacc2-9142-49d3-a23c-a62ba15e855e nodeName:}" failed. No retries permitted until 2026-05-28 00:12:20.854993681 +0000 UTC m=+50.182659767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert") pod "ingress-canary-4sppr" (UID: "da2bacc2-9142-49d3-a23c-a62ba15e855e") : secret "canary-serving-cert" not found
May 28 00:12:13.915434 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:13.915401    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"]
May 28 00:12:13.917225 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:13.917208    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"
May 28 00:12:13.919850 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:13.919824    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 28 00:12:13.919850 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:13.919847    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cjrv2\""
May 28 00:12:13.920019 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:13.919837    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 28 00:12:13.920135 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:13.920121    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 28 00:12:13.920196 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:13.920155    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 28 00:12:13.926161 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:13.926142    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"]
May 28 00:12:14.063161 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:14.063131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80337469-73b3-484b-b1f1-ce8944213baf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f94446f44-h8lwp\" (UID: \"80337469-73b3-484b-b1f1-ce8944213baf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"
May 28 00:12:14.063161 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:14.063162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x4dm\" (UniqueName: \"kubernetes.io/projected/80337469-73b3-484b-b1f1-ce8944213baf-kube-api-access-2x4dm\") pod \"managed-serviceaccount-addon-agent-6f94446f44-h8lwp\" (UID: \"80337469-73b3-484b-b1f1-ce8944213baf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"
May 28 00:12:14.163792 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:14.163760    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80337469-73b3-484b-b1f1-ce8944213baf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f94446f44-h8lwp\" (UID: \"80337469-73b3-484b-b1f1-ce8944213baf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"
May 28 00:12:14.163792 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:14.163794    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2x4dm\" (UniqueName: \"kubernetes.io/projected/80337469-73b3-484b-b1f1-ce8944213baf-kube-api-access-2x4dm\") pod \"managed-serviceaccount-addon-agent-6f94446f44-h8lwp\" (UID: \"80337469-73b3-484b-b1f1-ce8944213baf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"
May 28 00:12:14.167290 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:14.167227    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80337469-73b3-484b-b1f1-ce8944213baf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f94446f44-h8lwp\" (UID: \"80337469-73b3-484b-b1f1-ce8944213baf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"
May 28 00:12:14.174592 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:14.174569    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x4dm\" (UniqueName: \"kubernetes.io/projected/80337469-73b3-484b-b1f1-ce8944213baf-kube-api-access-2x4dm\") pod \"managed-serviceaccount-addon-agent-6f94446f44-h8lwp\" (UID: \"80337469-73b3-484b-b1f1-ce8944213baf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"
May 28 00:12:14.234594 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:14.234570    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"
May 28 00:12:14.359164 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:14.359127    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp"]
May 28 00:12:14.362315 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:12:14.362286    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80337469_73b3_484b_b1f1_ce8944213baf.slice/crio-81e11fed8f309ddf4b3ece723a3d3d2083ef9e931ce193464e59a64aae06e743 WatchSource:0}: Error finding container 81e11fed8f309ddf4b3ece723a3d3d2083ef9e931ce193464e59a64aae06e743: Status 404 returned error can't find the container with id 81e11fed8f309ddf4b3ece723a3d3d2083ef9e931ce193464e59a64aae06e743
May 28 00:12:14.553339 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:14.553308    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp" event={"ID":"80337469-73b3-484b-b1f1-ce8944213baf","Type":"ContainerStarted","Data":"81e11fed8f309ddf4b3ece723a3d3d2083ef9e931ce193464e59a64aae06e743"}
May 28 00:12:18.562509 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:18.562474    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp" event={"ID":"80337469-73b3-484b-b1f1-ce8944213baf","Type":"ContainerStarted","Data":"942f81159b4c016d12030a78be2652fa38dac2fbbbbd6d5921f468bd30f9baf1"}
May 28 00:12:18.577079 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:18.577025    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp" podStartSLOduration=2.11027635 podStartE2EDuration="5.577012393s" podCreationTimestamp="2026-05-28 00:12:13 +0000 UTC" firstStartedPulling="2026-05-28 00:12:14.364175573 +0000 UTC m=+43.691841659" lastFinishedPulling="2026-05-28 00:12:17.830911616 +0000 UTC m=+47.158577702" observedRunningTime="2026-05-28 00:12:18.57603163 +0000 UTC m=+47.903697739" watchObservedRunningTime="2026-05-28 00:12:18.577012393 +0000 UTC m=+47.904678500"
May 28 00:12:20.913046 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:20.913009    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:20.913046 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:20.913046    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:20.913437 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:20.913154    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:20.913437 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:20.913154    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:20.913437 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:20.913217    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls podName:200a652c-a7c2-4aee-8767-26801182f407 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:36.913199664 +0000 UTC m=+66.240865749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls") pod "dns-default-nkppb" (UID: "200a652c-a7c2-4aee-8767-26801182f407") : secret "dns-default-metrics-tls" not found
May 28 00:12:20.913437 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:20.913234    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert podName:da2bacc2-9142-49d3-a23c-a62ba15e855e nodeName:}" failed. No retries permitted until 2026-05-28 00:12:36.913226278 +0000 UTC m=+66.240892363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert") pod "ingress-canary-4sppr" (UID: "da2bacc2-9142-49d3-a23c-a62ba15e855e") : secret "canary-serving-cert" not found
May 28 00:12:30.526925 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:30.526899    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4zhg"
May 28 00:12:36.919420 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:36.919380    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:12:36.919420 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:36.919424    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:12:36.919857 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:36.919441    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:12:36.919857 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:36.919549    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:36.919857 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:36.919561    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:36.919857 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:36.919622    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls podName:200a652c-a7c2-4aee-8767-26801182f407 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:08.919606624 +0000 UTC m=+98.247272710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls") pod "dns-default-nkppb" (UID: "200a652c-a7c2-4aee-8767-26801182f407") : secret "dns-default-metrics-tls" not found
May 28 00:12:36.919857 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:36.919636    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert podName:da2bacc2-9142-49d3-a23c-a62ba15e855e nodeName:}" failed. No retries permitted until 2026-05-28 00:13:08.919630216 +0000 UTC m=+98.247296302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert") pod "ingress-canary-4sppr" (UID: "da2bacc2-9142-49d3-a23c-a62ba15e855e") : secret "canary-serving-cert" not found
May 28 00:12:36.921916 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:36.921901    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 00:12:36.930300 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:36.930282    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 00:12:36.930347 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:12:36.930338    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs podName:c0d3ce5a-3b25-4284-8946-bbb3e5820ce6 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:40.930323697 +0000 UTC m=+130.257989783 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs") pod "network-metrics-daemon-2dnl7" (UID: "c0d3ce5a-3b25-4284-8946-bbb3e5820ce6") : secret "metrics-daemon-secret" not found
May 28 00:12:37.020715 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:37.020683    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:12:37.023024 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:37.023010    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 00:12:37.033659 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:37.033637    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 00:12:37.044641 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:37.044618    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft8bw\" (UniqueName: \"kubernetes.io/projected/5dc8d16d-e5f7-410b-b433-0e30e23fd836-kube-api-access-ft8bw\") pod \"network-check-target-4sgrj\" (UID: \"5dc8d16d-e5f7-410b-b433-0e30e23fd836\") " pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:12:37.077859 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:37.077832    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q24q2\""
May 28 00:12:37.086673 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:37.086653    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:12:37.204287 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:37.204193    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4sgrj"]
May 28 00:12:37.207487 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:12:37.207461    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc8d16d_e5f7_410b_b433_0e30e23fd836.slice/crio-c598462f02cec825ad65f7e91ddfd312e7386cc79af3a277f78f78b94d9196c2 WatchSource:0}: Error finding container c598462f02cec825ad65f7e91ddfd312e7386cc79af3a277f78f78b94d9196c2: Status 404 returned error can't find the container with id c598462f02cec825ad65f7e91ddfd312e7386cc79af3a277f78f78b94d9196c2
May 28 00:12:37.600072 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:37.600028    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4sgrj" event={"ID":"5dc8d16d-e5f7-410b-b433-0e30e23fd836","Type":"ContainerStarted","Data":"c598462f02cec825ad65f7e91ddfd312e7386cc79af3a277f78f78b94d9196c2"}
May 28 00:12:40.607275 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:40.607235    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4sgrj" event={"ID":"5dc8d16d-e5f7-410b-b433-0e30e23fd836","Type":"ContainerStarted","Data":"67e4318ef5be9ce3191d313c3e2381c435d12fdf188fa6738a2f149ed028d101"}
May 28 00:12:40.607763 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:40.607351    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:12:40.622563 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:12:40.622504    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4sgrj" podStartSLOduration=67.061267375 podStartE2EDuration="1m9.622492551s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:12:37.209358929 +0000 UTC m=+66.537025022" lastFinishedPulling="2026-05-28 00:12:39.7705841 +0000 UTC m=+69.098250198" observedRunningTime="2026-05-28 00:12:40.622242294 +0000 UTC m=+69.949908402" watchObservedRunningTime="2026-05-28 00:12:40.622492551 +0000 UTC m=+69.950158659"
May 28 00:13:08.933948 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:08.933918    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:13:08.934303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:08.933956    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:13:08.934303 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:08.934052    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:13:08.934303 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:08.934079    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:13:08.934303 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:08.934110    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls podName:200a652c-a7c2-4aee-8767-26801182f407 nodeName:}" failed. No retries permitted until 2026-05-28 00:14:12.934097358 +0000 UTC m=+162.261763444 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls") pod "dns-default-nkppb" (UID: "200a652c-a7c2-4aee-8767-26801182f407") : secret "dns-default-metrics-tls" not found
May 28 00:13:08.934303 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:08.934159    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert podName:da2bacc2-9142-49d3-a23c-a62ba15e855e nodeName:}" failed. No retries permitted until 2026-05-28 00:14:12.934137623 +0000 UTC m=+162.261803712 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert") pod "ingress-canary-4sppr" (UID: "da2bacc2-9142-49d3-a23c-a62ba15e855e") : secret "canary-serving-cert" not found
May 28 00:13:11.611320 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:11.611283    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4sgrj"
May 28 00:13:34.369997 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.369963    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b8b555b99-htvmk"]
May 28 00:13:34.372313 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.372296    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.375552 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.375505    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 28 00:13:34.375666 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.375553    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 28 00:13:34.375666 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.375510    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 28 00:13:34.375813 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.375800    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-srcnj\""
May 28 00:13:34.382104 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.382086    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 28 00:13:34.386414 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.386390    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b8b555b99-htvmk"]
May 28 00:13:34.465145 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.465115    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn"]
May 28 00:13:34.467034 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.467015    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-74d5c8c648-2jk26"]
May 28 00:13:34.467170 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.467151    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn"
May 28 00:13:34.468961 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.468943    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-99bjj\""
May 28 00:13:34.469060 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.469002    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.471317 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.471303    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 28 00:13:34.471555 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.471537    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 28 00:13:34.471652 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.471562    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bbm85\""
May 28 00:13:34.471652 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.471562    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 28 00:13:34.471812 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.471800    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 28 00:13:34.472045 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.472030    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 28 00:13:34.472106 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.472057    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 28 00:13:34.483300 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.483283    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn"]
May 28 00:13:34.486542 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.486505    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74d5c8c648-2jk26"]
May 28 00:13:34.522508 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.522483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-bound-sa-token\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.522606 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.522532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-registry-certificates\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.522606 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.522561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-installation-pull-secrets\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.522606 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.522593    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fw2z\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-kube-api-access-8fw2z\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.522714 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.522610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-image-registry-private-configuration\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.522714 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.522636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.522714 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.522660    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c52f692-08d0-417d-bb51-51beb1c85510-ca-trust-extracted\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.522714 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.522676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-trusted-ca\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.566037 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.566014    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"]
May 28 00:13:34.567770 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.567756    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:34.569622 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.569604    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 28 00:13:34.569719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.569624    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mtqfl\""
May 28 00:13:34.569719 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.569624    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 28 00:13:34.569977 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.569961    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 28 00:13:34.570357 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.570341    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 28 00:13:34.576987 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.576968    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"]
May 28 00:13:34.623482 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623425    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-bound-sa-token\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.623482 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623458    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmzv\" (UniqueName: \"kubernetes.io/projected/bd4ec258-9332-443d-b262-f1b21d0e9337-kube-api-access-djmzv\") pod \"network-check-source-7f44b6dd87-66fkn\" (UID: \"bd4ec258-9332-443d-b262-f1b21d0e9337\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn"
May 28 00:13:34.623641 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfq4r\" (UniqueName: \"kubernetes.io/projected/207418e2-9809-45a0-85a1-9baaf64716b8-kube-api-access-hfq4r\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.623641 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623548    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-registry-certificates\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.623641 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623578    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-installation-pull-secrets\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.623641 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623604    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8fw2z\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-kube-api-access-8fw2z\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.623641 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623629    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.623894 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623657    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-image-registry-private-configuration\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.623894 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623684    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-stats-auth\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.623894 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623712    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.623894 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623739    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c52f692-08d0-417d-bb51-51beb1c85510-ca-trust-extracted\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.623894 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623771    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-trusted-ca\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.623894 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:34.623809    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:13:34.623894 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:34.623829    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b8b555b99-htvmk: secret "image-registry-tls" not found
May 28 00:13:34.623894 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-default-certificate\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.623894 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:34.623886    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls podName:5c52f692-08d0-417d-bb51-51beb1c85510 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:35.123866627 +0000 UTC m=+124.451532715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls") pod "image-registry-b8b555b99-htvmk" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510") : secret "image-registry-tls" not found
May 28 00:13:34.624256 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.623912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.624256 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.624127    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c52f692-08d0-417d-bb51-51beb1c85510-ca-trust-extracted\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.624256 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.624139    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-registry-certificates\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.624754 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.624725    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-trusted-ca\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.626161 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.626143    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-installation-pull-secrets\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.626283 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.626268    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-image-registry-private-configuration\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.631751 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.631731    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-bound-sa-token\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.631917 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.631899    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fw2z\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-kube-api-access-8fw2z\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:34.667774 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.667750    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"]
May 28 00:13:34.669901 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.669887    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.671793 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.671768    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 28 00:13:34.671887 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.671852    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ndcxd\""
May 28 00:13:34.671887 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.671860    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 28 00:13:34.671887 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.671882    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 28 00:13:34.672021 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.671912    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 28 00:13:34.677775 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.677755    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"]
May 28 00:13:34.724604 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.724583    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hfq4r\" (UniqueName: \"kubernetes.io/projected/207418e2-9809-45a0-85a1-9baaf64716b8-kube-api-access-hfq4r\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.724704 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.724618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:34.724704 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.724644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfm46\" (UniqueName: \"kubernetes.io/projected/b5068452-b413-411d-a928-b8e91a4b942a-kube-api-access-hfm46\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:34.724704 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.724680    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.724864 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.724714    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-stats-auth\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.724864 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:34.724798    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:35.224777694 +0000 UTC m=+124.552443804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : configmap references non-existent config key: service-ca.crt
May 28 00:13:34.724864 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.724847    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-default-certificate\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.725016 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.724889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b5068452-b413-411d-a928-b8e91a4b942a-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:34.725016 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.724921    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.725016 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.724954    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-djmzv\" (UniqueName: \"kubernetes.io/projected/bd4ec258-9332-443d-b262-f1b21d0e9337-kube-api-access-djmzv\") pod \"network-check-source-7f44b6dd87-66fkn\" (UID: \"bd4ec258-9332-443d-b262-f1b21d0e9337\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn"
May 28 00:13:34.725157 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:34.725020    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 00:13:34.725157 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:34.725075    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:35.225059764 +0000 UTC m=+124.552725857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : secret "router-metrics-certs-default" not found
May 28 00:13:34.727144 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.727125    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-default-certificate\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.727270 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.727251    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-stats-auth\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.734590 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.734570    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmzv\" (UniqueName: \"kubernetes.io/projected/bd4ec258-9332-443d-b262-f1b21d0e9337-kube-api-access-djmzv\") pod \"network-check-source-7f44b6dd87-66fkn\" (UID: \"bd4ec258-9332-443d-b262-f1b21d0e9337\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn"
May 28 00:13:34.734689 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.734672    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfq4r\" (UniqueName: \"kubernetes.io/projected/207418e2-9809-45a0-85a1-9baaf64716b8-kube-api-access-hfq4r\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:34.776737 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.776717    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn"
May 28 00:13:34.825882 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.825850    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b5068452-b413-411d-a928-b8e91a4b942a-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:34.825985 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.825914    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abfdf3bc-d254-45a0-b65c-5a67c4490eb8-serving-cert\") pod \"service-ca-operator-57b5b68b8-m7mzw\" (UID: \"abfdf3bc-d254-45a0-b65c-5a67c4490eb8\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.825985 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.825970    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:34.826090 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.825997    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hfm46\" (UniqueName: \"kubernetes.io/projected/b5068452-b413-411d-a928-b8e91a4b942a-kube-api-access-hfm46\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:34.826090 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.826027    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/abfdf3bc-d254-45a0-b65c-5a67c4490eb8-config\") pod \"service-ca-operator-57b5b68b8-m7mzw\" (UID: \"abfdf3bc-d254-45a0-b65c-5a67c4490eb8\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.826186 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:34.826089    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 00:13:34.826186 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:34.826176    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls podName:b5068452-b413-411d-a928-b8e91a4b942a nodeName:}" failed. No retries permitted until 2026-05-28 00:13:35.326153322 +0000 UTC m=+124.653819411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-fsmz8" (UID: "b5068452-b413-411d-a928-b8e91a4b942a") : secret "cluster-monitoring-operator-tls" not found
May 28 00:13:34.826287 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.826097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmpg\" (UniqueName: \"kubernetes.io/projected/abfdf3bc-d254-45a0-b65c-5a67c4490eb8-kube-api-access-qbmpg\") pod \"service-ca-operator-57b5b68b8-m7mzw\" (UID: \"abfdf3bc-d254-45a0-b65c-5a67c4490eb8\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.826484 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.826464    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b5068452-b413-411d-a928-b8e91a4b942a-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:34.834840 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.834815    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfm46\" (UniqueName: \"kubernetes.io/projected/b5068452-b413-411d-a928-b8e91a4b942a-kube-api-access-hfm46\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:34.891633 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.891564    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn"]
May 28 00:13:34.894740 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:13:34.894709    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4ec258_9332_443d_b262_f1b21d0e9337.slice/crio-24c9b978323c4f480f154ace6d54853c48acbccb16c1e77d41000be975daf6b9 WatchSource:0}: Error finding container 24c9b978323c4f480f154ace6d54853c48acbccb16c1e77d41000be975daf6b9: Status 404 returned error can't find the container with id 24c9b978323c4f480f154ace6d54853c48acbccb16c1e77d41000be975daf6b9
May 28 00:13:34.927474 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.927450    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmpg\" (UniqueName: \"kubernetes.io/projected/abfdf3bc-d254-45a0-b65c-5a67c4490eb8-kube-api-access-qbmpg\") pod \"service-ca-operator-57b5b68b8-m7mzw\" (UID: \"abfdf3bc-d254-45a0-b65c-5a67c4490eb8\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.927629 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.927611    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abfdf3bc-d254-45a0-b65c-5a67c4490eb8-serving-cert\") pod \"service-ca-operator-57b5b68b8-m7mzw\" (UID: \"abfdf3bc-d254-45a0-b65c-5a67c4490eb8\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.927694 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.927657    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/abfdf3bc-d254-45a0-b65c-5a67c4490eb8-config\") pod \"service-ca-operator-57b5b68b8-m7mzw\" (UID: \"abfdf3bc-d254-45a0-b65c-5a67c4490eb8\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.928095 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.928076    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/abfdf3bc-d254-45a0-b65c-5a67c4490eb8-config\") pod \"service-ca-operator-57b5b68b8-m7mzw\" (UID: \"abfdf3bc-d254-45a0-b65c-5a67c4490eb8\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.929889 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.929850    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abfdf3bc-d254-45a0-b65c-5a67c4490eb8-serving-cert\") pod \"service-ca-operator-57b5b68b8-m7mzw\" (UID: \"abfdf3bc-d254-45a0-b65c-5a67c4490eb8\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.937057 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.937033    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmpg\" (UniqueName: \"kubernetes.io/projected/abfdf3bc-d254-45a0-b65c-5a67c4490eb8-kube-api-access-qbmpg\") pod \"service-ca-operator-57b5b68b8-m7mzw\" (UID: \"abfdf3bc-d254-45a0-b65c-5a67c4490eb8\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:34.979045 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:34.979021    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"
May 28 00:13:35.096333 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:35.096302    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw"]
May 28 00:13:35.100664 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:13:35.100631    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfdf3bc_d254_45a0_b65c_5a67c4490eb8.slice/crio-b575eed0b25fcbc34482817fccde83470b748249d280370244e97379847f59ff WatchSource:0}: Error finding container b575eed0b25fcbc34482817fccde83470b748249d280370244e97379847f59ff: Status 404 returned error can't find the container with id b575eed0b25fcbc34482817fccde83470b748249d280370244e97379847f59ff
May 28 00:13:35.130329 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:35.130305    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:35.130420 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:35.130412    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:13:35.130480 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:35.130424    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b8b555b99-htvmk: secret "image-registry-tls" not found
May 28 00:13:35.130480 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:35.130465    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls podName:5c52f692-08d0-417d-bb51-51beb1c85510 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:36.130450534 +0000 UTC m=+125.458116619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls") pod "image-registry-b8b555b99-htvmk" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510") : secret "image-registry-tls" not found
May 28 00:13:35.230658 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:35.230591    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:35.230658 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:35.230649    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:35.230812 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:35.230733    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 00:13:35.230812 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:35.230768    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:36.23075398 +0000 UTC m=+125.558420066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : configmap references non-existent config key: service-ca.crt
May 28 00:13:35.230812 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:35.230782    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:36.230776871 +0000 UTC m=+125.558442956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : secret "router-metrics-certs-default" not found
May 28 00:13:35.334981 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:35.331806    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:35.334981 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:35.332132    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 00:13:35.334981 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:35.332194    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls podName:b5068452-b413-411d-a928-b8e91a4b942a nodeName:}" failed. No retries permitted until 2026-05-28 00:13:36.332178341 +0000 UTC m=+125.659844431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-fsmz8" (UID: "b5068452-b413-411d-a928-b8e91a4b942a") : secret "cluster-monitoring-operator-tls" not found
May 28 00:13:35.710928 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:35.710887    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw" event={"ID":"abfdf3bc-d254-45a0-b65c-5a67c4490eb8","Type":"ContainerStarted","Data":"b575eed0b25fcbc34482817fccde83470b748249d280370244e97379847f59ff"}
May 28 00:13:35.712121 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:35.712094    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn" event={"ID":"bd4ec258-9332-443d-b262-f1b21d0e9337","Type":"ContainerStarted","Data":"86c8d3e627dada344a4b10563e5dff187b2d3040b95ee7d7f0368cdcffd9562e"}
May 28 00:13:35.712205 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:35.712128    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn" event={"ID":"bd4ec258-9332-443d-b262-f1b21d0e9337","Type":"ContainerStarted","Data":"24c9b978323c4f480f154ace6d54853c48acbccb16c1e77d41000be975daf6b9"}
May 28 00:13:36.136393 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:36.136355    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:36.136575 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:36.136488    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:13:36.136575 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:36.136508    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b8b555b99-htvmk: secret "image-registry-tls" not found
May 28 00:13:36.136651 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:36.136583    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls podName:5c52f692-08d0-417d-bb51-51beb1c85510 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:38.136568045 +0000 UTC m=+127.464234136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls") pod "image-registry-b8b555b99-htvmk" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510") : secret "image-registry-tls" not found
May 28 00:13:36.237331 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:36.237302    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:36.237451 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:36.237358    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:36.237451 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:36.237447    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:38.237434096 +0000 UTC m=+127.565100182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : configmap references non-existent config key: service-ca.crt
May 28 00:13:36.237550 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:36.237457    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 00:13:36.237550 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:36.237510    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:38.237493575 +0000 UTC m=+127.565159664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : secret "router-metrics-certs-default" not found
May 28 00:13:36.338009 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:36.337977    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:36.338161 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:36.338124    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 00:13:36.338217 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:36.338190    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls podName:b5068452-b413-411d-a928-b8e91a4b942a nodeName:}" failed. No retries permitted until 2026-05-28 00:13:38.338171823 +0000 UTC m=+127.665837923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-fsmz8" (UID: "b5068452-b413-411d-a928-b8e91a4b942a") : secret "cluster-monitoring-operator-tls" not found
May 28 00:13:38.157271 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:38.157225    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:38.157756 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:38.157395    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:13:38.157756 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:38.157420    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b8b555b99-htvmk: secret "image-registry-tls" not found
May 28 00:13:38.157756 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:38.157492    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls podName:5c52f692-08d0-417d-bb51-51beb1c85510 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:42.157470926 +0000 UTC m=+131.485137033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls") pod "image-registry-b8b555b99-htvmk" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510") : secret "image-registry-tls" not found
May 28 00:13:38.257666 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:38.257640    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:38.257766 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:38.257753    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:38.257806 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:38.257771    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:42.257755036 +0000 UTC m=+131.585421122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : configmap references non-existent config key: service-ca.crt
May 28 00:13:38.257853 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:38.257841    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 00:13:38.257903 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:38.257894    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:42.25787846 +0000 UTC m=+131.585544552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : secret "router-metrics-certs-default" not found
May 28 00:13:38.359111 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:38.359075    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:38.359273 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:38.359219    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 00:13:38.359348 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:38.359281    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls podName:b5068452-b413-411d-a928-b8e91a4b942a nodeName:}" failed. No retries permitted until 2026-05-28 00:13:42.359266073 +0000 UTC m=+131.686932159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-fsmz8" (UID: "b5068452-b413-411d-a928-b8e91a4b942a") : secret "cluster-monitoring-operator-tls" not found
May 28 00:13:38.717881 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:38.717848    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw" event={"ID":"abfdf3bc-d254-45a0-b65c-5a67c4490eb8","Type":"ContainerStarted","Data":"3374430f39b0e04556d5f3a9f79f866599d2c1cb196567b4cb6bd0b044dcf5a9"}
May 28 00:13:38.731235 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:38.731172    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw" podStartSLOduration=1.635531235 podStartE2EDuration="4.731156442s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="2026-05-28 00:13:35.102671897 +0000 UTC m=+124.430337989" lastFinishedPulling="2026-05-28 00:13:38.198297106 +0000 UTC m=+127.525963196" observedRunningTime="2026-05-28 00:13:38.730789774 +0000 UTC m=+128.058455882" watchObservedRunningTime="2026-05-28 00:13:38.731156442 +0000 UTC m=+128.058822550"
May 28 00:13:38.731999 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:38.731972    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-66fkn" podStartSLOduration=4.73196354 podStartE2EDuration="4.73196354s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:13:35.72971956 +0000 UTC m=+125.057385668" watchObservedRunningTime="2026-05-28 00:13:38.73196354 +0000 UTC m=+128.059629647"
May 28 00:13:40.981192 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:40.981150    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:13:40.981777 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:40.981292    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 00:13:40.981777 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:40.981356    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs podName:c0d3ce5a-3b25-4284-8946-bbb3e5820ce6 nodeName:}" failed. No retries permitted until 2026-05-28 00:15:42.981340731 +0000 UTC m=+252.309006821 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs") pod "network-metrics-daemon-2dnl7" (UID: "c0d3ce5a-3b25-4284-8946-bbb3e5820ce6") : secret "metrics-daemon-secret" not found
May 28 00:13:42.191227 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:42.191189    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:42.191606 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:42.191329    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:13:42.191606 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:42.191348    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b8b555b99-htvmk: secret "image-registry-tls" not found
May 28 00:13:42.191606 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:42.191404    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls podName:5c52f692-08d0-417d-bb51-51beb1c85510 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:50.191388595 +0000 UTC m=+139.519054681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls") pod "image-registry-b8b555b99-htvmk" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510") : secret "image-registry-tls" not found
May 28 00:13:42.292381 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:42.292356    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:42.292461 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:42.292428    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:42.292575 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:42.292555    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:50.292509597 +0000 UTC m=+139.620175685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : configmap references non-existent config key: service-ca.crt
May 28 00:13:42.292619 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:42.292556    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 00:13:42.292619 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:42.292611    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:50.292603061 +0000 UTC m=+139.620269148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : secret "router-metrics-certs-default" not found
May 28 00:13:42.393425 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:42.393391    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:42.393644 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:42.393573    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 00:13:42.393644 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:42.393640    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls podName:b5068452-b413-411d-a928-b8e91a4b942a nodeName:}" failed. No retries permitted until 2026-05-28 00:13:50.393622363 +0000 UTC m=+139.721288467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-fsmz8" (UID: "b5068452-b413-411d-a928-b8e91a4b942a") : secret "cluster-monitoring-operator-tls" not found
May 28 00:13:42.394184 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:42.394165    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7qxzh_bbde4ba0-8948-4141-93a0-c0ff1aa62641/dns-node-resolver/0.log"
May 28 00:13:43.795172 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:43.795147    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qfqhg_9315e6c2-c11e-4ee7-a5fd-5f57e12772a6/node-ca/0.log"
May 28 00:13:50.251967 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.251933    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:50.254366 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.254335    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") pod \"image-registry-b8b555b99-htvmk\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") " pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:50.281156 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.281133    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:50.352984 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.352954    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:50.353113 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.353034    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:13:50.353156 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:50.353146    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 00:13:50.353192 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:50.353169    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:14:06.353144644 +0000 UTC m=+155.680810745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : configmap references non-existent config key: service-ca.crt
May 28 00:13:50.353232 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:50.353201    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs podName:207418e2-9809-45a0-85a1-9baaf64716b8 nodeName:}" failed. No retries permitted until 2026-05-28 00:14:06.353188664 +0000 UTC m=+155.680854756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs") pod "router-default-74d5c8c648-2jk26" (UID: "207418e2-9809-45a0-85a1-9baaf64716b8") : secret "router-metrics-certs-default" not found
May 28 00:13:50.405983 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.405953    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b8b555b99-htvmk"]
May 28 00:13:50.408715 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:13:50.408688    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c52f692_08d0_417d_bb51_51beb1c85510.slice/crio-94e05978519be7caa2750cf9ecefe185c5c1416216447baf2dfbbdab8e64a4cd WatchSource:0}: Error finding container 94e05978519be7caa2750cf9ecefe185c5c1416216447baf2dfbbdab8e64a4cd: Status 404 returned error can't find the container with id 94e05978519be7caa2750cf9ecefe185c5c1416216447baf2dfbbdab8e64a4cd
May 28 00:13:50.453925 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.453903    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:13:50.454036 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:50.454024    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 00:13:50.454083 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:13:50.454074    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls podName:b5068452-b413-411d-a928-b8e91a4b942a nodeName:}" failed. No retries permitted until 2026-05-28 00:14:06.45406004 +0000 UTC m=+155.781726126 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-fsmz8" (UID: "b5068452-b413-411d-a928-b8e91a4b942a") : secret "cluster-monitoring-operator-tls" not found
May 28 00:13:50.742116 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.742072    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b8b555b99-htvmk" event={"ID":"5c52f692-08d0-417d-bb51-51beb1c85510","Type":"ContainerStarted","Data":"37d80e50a98bdc292f367e7d5b9f2664cdb6169654a22c60e6884e2a30178c49"}
May 28 00:13:50.742116 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.742120    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b8b555b99-htvmk" event={"ID":"5c52f692-08d0-417d-bb51-51beb1c85510","Type":"ContainerStarted","Data":"94e05978519be7caa2750cf9ecefe185c5c1416216447baf2dfbbdab8e64a4cd"}
May 28 00:13:50.742315 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.742196    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:13:50.761179 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:13:50.761135    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b8b555b99-htvmk" podStartSLOduration=16.761122606 podStartE2EDuration="16.761122606s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:13:50.760733081 +0000 UTC m=+140.088399188" watchObservedRunningTime="2026-05-28 00:13:50.761122606 +0000 UTC m=+140.088788713"
May 28 00:14:01.696317 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.696281    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-mtcq5"]
May 28 00:14:01.699278 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.699257    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-mtcq5"
May 28 00:14:01.701316 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.701291    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b8b555b99-htvmk"]
May 28 00:14:01.701945 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.701922    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 28 00:14:01.702058 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.701993    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 28 00:14:01.702058 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.702031    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-k2sn9\""
May 28 00:14:01.709001 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.708977    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-mtcq5"]
May 28 00:14:01.735983 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.735963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8hss\" (UniqueName: \"kubernetes.io/projected/2d6f4edf-5055-40c8-a687-08f1753d6bc0-kube-api-access-l8hss\") pod \"downloads-57bd96b888-mtcq5\" (UID: \"2d6f4edf-5055-40c8-a687-08f1753d6bc0\") " pod="openshift-console/downloads-57bd96b888-mtcq5"
May 28 00:14:01.739772 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.739752    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-577655c79f-9zmnl"]
May 28 00:14:01.741412 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.741398    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.753947 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.753925    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577655c79f-9zmnl"]
May 28 00:14:01.762206 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.762185    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ftvgh"]
May 28 00:14:01.764341 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.764324    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.770195 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.770174    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 28 00:14:01.770535 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.770496    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 28 00:14:01.770683 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.770665    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rdsbx\""
May 28 00:14:01.770730 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.770717    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 28 00:14:01.771313 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.771297    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 28 00:14:01.783183 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.783162    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ftvgh"]
May 28 00:14:01.836311 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66068cd5-441a-4aac-819d-47ededa2a63f-registry-tls\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.836311 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836314    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-crio-socket\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.836503 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66068cd5-441a-4aac-819d-47ededa2a63f-ca-trust-extracted\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.836503 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66068cd5-441a-4aac-819d-47ededa2a63f-installation-pull-secrets\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.836503 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqc6x\" (UniqueName: \"kubernetes.io/projected/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-kube-api-access-lqc6x\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.836631 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6xl\" (UniqueName: \"kubernetes.io/projected/66068cd5-441a-4aac-819d-47ededa2a63f-kube-api-access-8x6xl\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.836631 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836566    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l8hss\" (UniqueName: \"kubernetes.io/projected/2d6f4edf-5055-40c8-a687-08f1753d6bc0-kube-api-access-l8hss\") pod \"downloads-57bd96b888-mtcq5\" (UID: \"2d6f4edf-5055-40c8-a687-08f1753d6bc0\") " pod="openshift-console/downloads-57bd96b888-mtcq5"
May 28 00:14:01.836631 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.836631 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66068cd5-441a-4aac-819d-47ededa2a63f-image-registry-private-configuration\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.836631 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66068cd5-441a-4aac-819d-47ededa2a63f-registry-certificates\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.836800 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836664    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-data-volume\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.836800 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66068cd5-441a-4aac-819d-47ededa2a63f-bound-sa-token\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.836800 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66068cd5-441a-4aac-819d-47ededa2a63f-trusted-ca\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.836800 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.836743    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.844488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.844467    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8hss\" (UniqueName: \"kubernetes.io/projected/2d6f4edf-5055-40c8-a687-08f1753d6bc0-kube-api-access-l8hss\") pod \"downloads-57bd96b888-mtcq5\" (UID: \"2d6f4edf-5055-40c8-a687-08f1753d6bc0\") " pod="openshift-console/downloads-57bd96b888-mtcq5"
May 28 00:14:01.937776 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.937736    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lqc6x\" (UniqueName: \"kubernetes.io/projected/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-kube-api-access-lqc6x\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.937963 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.937799    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6xl\" (UniqueName: \"kubernetes.io/projected/66068cd5-441a-4aac-819d-47ededa2a63f-kube-api-access-8x6xl\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.937963 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.937830    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.937963 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.937856    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66068cd5-441a-4aac-819d-47ededa2a63f-image-registry-private-configuration\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.937963 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.937878    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66068cd5-441a-4aac-819d-47ededa2a63f-registry-certificates\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.938162 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938018    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-data-volume\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.938162 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938054    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66068cd5-441a-4aac-819d-47ededa2a63f-bound-sa-token\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.938162 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938084    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66068cd5-441a-4aac-819d-47ededa2a63f-trusted-ca\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.938162 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938119    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.938359 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938178    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66068cd5-441a-4aac-819d-47ededa2a63f-registry-tls\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.938359 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938337    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-crio-socket\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.938454 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938374    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66068cd5-441a-4aac-819d-47ededa2a63f-ca-trust-extracted\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.938454 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938406    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66068cd5-441a-4aac-819d-47ededa2a63f-installation-pull-secrets\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.938454 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938430    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-crio-socket\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.938661 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938553    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.938661 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938373    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-data-volume\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.938777 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938757    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66068cd5-441a-4aac-819d-47ededa2a63f-ca-trust-extracted\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.938833 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.938796    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66068cd5-441a-4aac-819d-47ededa2a63f-registry-certificates\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.939187 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.939162    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66068cd5-441a-4aac-819d-47ededa2a63f-trusted-ca\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.940614 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.940588    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66068cd5-441a-4aac-819d-47ededa2a63f-image-registry-private-configuration\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.941053 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.941028    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66068cd5-441a-4aac-819d-47ededa2a63f-registry-tls\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.941287 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.941263    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.941338 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.941324    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66068cd5-441a-4aac-819d-47ededa2a63f-installation-pull-secrets\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.948905 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.948855    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66068cd5-441a-4aac-819d-47ededa2a63f-bound-sa-token\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:01.949075 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.949054    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqc6x\" (UniqueName: \"kubernetes.io/projected/e4375a32-e2b1-4f59-8e8e-1cfa96facde4-kube-api-access-lqc6x\") pod \"insights-runtime-extractor-ftvgh\" (UID: \"e4375a32-e2b1-4f59-8e8e-1cfa96facde4\") " pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:01.949303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:01.949286    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6xl\" (UniqueName: \"kubernetes.io/projected/66068cd5-441a-4aac-819d-47ededa2a63f-kube-api-access-8x6xl\") pod \"image-registry-577655c79f-9zmnl\" (UID: \"66068cd5-441a-4aac-819d-47ededa2a63f\") " pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:02.008532 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.008491    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-mtcq5"
May 28 00:14:02.049301 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.049272    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:02.072260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.072224    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ftvgh"
May 28 00:14:02.149649 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.149543    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-mtcq5"]
May 28 00:14:02.152933 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:14:02.152892    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6f4edf_5055_40c8_a687_08f1753d6bc0.slice/crio-6e61c849687fc6bb1340aba8641f1461a059a9defd1b59671153e62035e8d925 WatchSource:0}: Error finding container 6e61c849687fc6bb1340aba8641f1461a059a9defd1b59671153e62035e8d925: Status 404 returned error can't find the container with id 6e61c849687fc6bb1340aba8641f1461a059a9defd1b59671153e62035e8d925
May 28 00:14:02.192796 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.192766    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577655c79f-9zmnl"]
May 28 00:14:02.195671 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:14:02.195636    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66068cd5_441a_4aac_819d_47ededa2a63f.slice/crio-11393ec636b026af39aee2c1b1dc25db17272cce108584cbfa041930f5606bdc WatchSource:0}: Error finding container 11393ec636b026af39aee2c1b1dc25db17272cce108584cbfa041930f5606bdc: Status 404 returned error can't find the container with id 11393ec636b026af39aee2c1b1dc25db17272cce108584cbfa041930f5606bdc
May 28 00:14:02.213539 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.213495    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ftvgh"]
May 28 00:14:02.219403 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:14:02.219382    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4375a32_e2b1_4f59_8e8e_1cfa96facde4.slice/crio-75f3aabd13656d52327069f911abba6fcbe796e60625cff54755c6054014b7eb WatchSource:0}: Error finding container 75f3aabd13656d52327069f911abba6fcbe796e60625cff54755c6054014b7eb: Status 404 returned error can't find the container with id 75f3aabd13656d52327069f911abba6fcbe796e60625cff54755c6054014b7eb
May 28 00:14:02.768198 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.768163    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ftvgh" event={"ID":"e4375a32-e2b1-4f59-8e8e-1cfa96facde4","Type":"ContainerStarted","Data":"bd790aca5b4f502cc1d1c2899724dd173a2bef82b56ef719f10cb30ed02b4edb"}
May 28 00:14:02.768198 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.768200    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ftvgh" event={"ID":"e4375a32-e2b1-4f59-8e8e-1cfa96facde4","Type":"ContainerStarted","Data":"75f3aabd13656d52327069f911abba6fcbe796e60625cff54755c6054014b7eb"}
May 28 00:14:02.769202 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.769180    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-mtcq5" event={"ID":"2d6f4edf-5055-40c8-a687-08f1753d6bc0","Type":"ContainerStarted","Data":"6e61c849687fc6bb1340aba8641f1461a059a9defd1b59671153e62035e8d925"}
May 28 00:14:02.770268 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.770246    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577655c79f-9zmnl" event={"ID":"66068cd5-441a-4aac-819d-47ededa2a63f","Type":"ContainerStarted","Data":"ba42a895c6c841f28a59a94afd92db405978a76ad9b017c77159fdae550be2a3"}
May 28 00:14:02.770325 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.770277    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577655c79f-9zmnl" event={"ID":"66068cd5-441a-4aac-819d-47ededa2a63f","Type":"ContainerStarted","Data":"11393ec636b026af39aee2c1b1dc25db17272cce108584cbfa041930f5606bdc"}
May 28 00:14:02.770389 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.770373    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:02.789287 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:02.789238    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-577655c79f-9zmnl" podStartSLOduration=1.789223932 podStartE2EDuration="1.789223932s" podCreationTimestamp="2026-05-28 00:14:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:14:02.788202506 +0000 UTC m=+152.115868614" watchObservedRunningTime="2026-05-28 00:14:02.789223932 +0000 UTC m=+152.116890037"
May 28 00:14:03.774171 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:03.774129    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ftvgh" event={"ID":"e4375a32-e2b1-4f59-8e8e-1cfa96facde4","Type":"ContainerStarted","Data":"3a7abe5e5189774115fd6ec1d6a5f8dd455cb99e3c40e52991cd95d800beb4c0"}
May 28 00:14:05.781218 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:05.781180    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ftvgh" event={"ID":"e4375a32-e2b1-4f59-8e8e-1cfa96facde4","Type":"ContainerStarted","Data":"07aeb6c69b0525236d39834372740cc95e84f30f0e97fa1d812d4a3131de8a4e"}
May 28 00:14:05.799395 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:05.799343    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ftvgh" podStartSLOduration=2.237498609 podStartE2EDuration="4.799329116s" podCreationTimestamp="2026-05-28 00:14:01 +0000 UTC" firstStartedPulling="2026-05-28 00:14:02.262324014 +0000 UTC m=+151.589990113" lastFinishedPulling="2026-05-28 00:14:04.824154535 +0000 UTC m=+154.151820620" observedRunningTime="2026-05-28 00:14:05.798472475 +0000 UTC m=+155.126138585" watchObservedRunningTime="2026-05-28 00:14:05.799329116 +0000 UTC m=+155.126995223"
May 28 00:14:06.378534 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.378479    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:14:06.378720 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.378597    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:14:06.379167 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.379143    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207418e2-9809-45a0-85a1-9baaf64716b8-service-ca-bundle\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:14:06.381303 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.381275    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207418e2-9809-45a0-85a1-9baaf64716b8-metrics-certs\") pod \"router-default-74d5c8c648-2jk26\" (UID: \"207418e2-9809-45a0-85a1-9baaf64716b8\") " pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:14:06.479029 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.479001    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:14:06.481739 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.481697    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5068452-b413-411d-a928-b8e91a4b942a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-fsmz8\" (UID: \"b5068452-b413-411d-a928-b8e91a4b942a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:14:06.581684 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.581645    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:14:06.676253 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.676199    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"
May 28 00:14:06.715751 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.715712    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74d5c8c648-2jk26"]
May 28 00:14:06.718129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:14:06.718098    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207418e2_9809_45a0_85a1_9baaf64716b8.slice/crio-0f0bb85a1cfb354e7509a13676c1bcbf96498c5c9ffb3480c6d2f0e8e9bf3d18 WatchSource:0}: Error finding container 0f0bb85a1cfb354e7509a13676c1bcbf96498c5c9ffb3480c6d2f0e8e9bf3d18: Status 404 returned error can't find the container with id 0f0bb85a1cfb354e7509a13676c1bcbf96498c5c9ffb3480c6d2f0e8e9bf3d18
May 28 00:14:06.786207 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.786154    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74d5c8c648-2jk26" event={"ID":"207418e2-9809-45a0-85a1-9baaf64716b8","Type":"ContainerStarted","Data":"0f0bb85a1cfb354e7509a13676c1bcbf96498c5c9ffb3480c6d2f0e8e9bf3d18"}
May 28 00:14:06.822030 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:06.821996    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8"]
May 28 00:14:06.826497 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:14:06.826464    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5068452_b413_411d_a928_b8e91a4b942a.slice/crio-197eaf7b2cb2c21d08b8c1ffbc70a69922d862a13ceb617f9de0b776c578e8f5 WatchSource:0}: Error finding container 197eaf7b2cb2c21d08b8c1ffbc70a69922d862a13ceb617f9de0b776c578e8f5: Status 404 returned error can't find the container with id 197eaf7b2cb2c21d08b8c1ffbc70a69922d862a13ceb617f9de0b776c578e8f5
May 28 00:14:07.791662 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:07.791601    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74d5c8c648-2jk26" event={"ID":"207418e2-9809-45a0-85a1-9baaf64716b8","Type":"ContainerStarted","Data":"1e22f1d0978def6e9ab4f256bf806721c617891f1ea1c484278df81e6089cd3a"}
May 28 00:14:07.793643 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:07.793599    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8" event={"ID":"b5068452-b413-411d-a928-b8e91a4b942a","Type":"ContainerStarted","Data":"197eaf7b2cb2c21d08b8c1ffbc70a69922d862a13ceb617f9de0b776c578e8f5"}
May 28 00:14:07.815037 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:07.814985    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-74d5c8c648-2jk26" podStartSLOduration=33.814967258 podStartE2EDuration="33.814967258s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:14:07.814113541 +0000 UTC m=+157.141779650" watchObservedRunningTime="2026-05-28 00:14:07.814967258 +0000 UTC m=+157.142633367"
May 28 00:14:08.102914 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:14:08.102826    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nkppb" podUID="200a652c-a7c2-4aee-8767-26801182f407"
May 28 00:14:08.116726 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:14:08.116688    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-4sppr" podUID="da2bacc2-9142-49d3-a23c-a62ba15e855e"
May 28 00:14:08.269528 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:14:08.269467    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2dnl7" podUID="c0d3ce5a-3b25-4284-8946-bbb3e5820ce6"
May 28 00:14:08.582574 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:08.582491    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:14:08.585345 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:08.585321    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:14:08.796054 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:08.796022    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nkppb"
May 28 00:14:08.796480 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:08.796270    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:14:08.797655 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:08.797633    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-74d5c8c648-2jk26"
May 28 00:14:09.800401 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:09.800354    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8" event={"ID":"b5068452-b413-411d-a928-b8e91a4b942a","Type":"ContainerStarted","Data":"76e818eb68b116a442fdfdb129b2aaa18713708eaecaec02276e06f5899db98f"}
May 28 00:14:09.816275 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:09.816221    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-fsmz8" podStartSLOduration=33.65743965 podStartE2EDuration="35.816204111s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="2026-05-28 00:14:06.828818052 +0000 UTC m=+156.156484141" lastFinishedPulling="2026-05-28 00:14:08.987582499 +0000 UTC m=+158.315248602" observedRunningTime="2026-05-28 00:14:09.815752384 +0000 UTC m=+159.143418505" watchObservedRunningTime="2026-05-28 00:14:09.816204111 +0000 UTC m=+159.143870221"
May 28 00:14:11.708846 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:11.708809    2560 patch_prober.go:28] interesting pod/image-registry-b8b555b99-htvmk container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 28 00:14:11.709255 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:11.708866    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-b8b555b99-htvmk" podUID="5c52f692-08d0-417d-bb51-51beb1c85510" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 28 00:14:12.939046 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:12.938942    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:14:12.939046 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:12.939002    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:14:12.941805 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:12.941778    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/200a652c-a7c2-4aee-8767-26801182f407-metrics-tls\") pod \"dns-default-nkppb\" (UID: \"200a652c-a7c2-4aee-8767-26801182f407\") " pod="openshift-dns/dns-default-nkppb"
May 28 00:14:12.941909 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:12.941851    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da2bacc2-9142-49d3-a23c-a62ba15e855e-cert\") pod \"ingress-canary-4sppr\" (UID: \"da2bacc2-9142-49d3-a23c-a62ba15e855e\") " pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:14:12.999725 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:12.999694    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzrg4\""
May 28 00:14:13.007696 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:13.007668    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nkppb"
May 28 00:14:18.967272 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:18.967243    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nkppb"]
May 28 00:14:18.978469 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:14:18.978442    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200a652c_a7c2_4aee_8767_26801182f407.slice/crio-2938c929f268dcee1294b4a934544cb8528f8a09f9e60aec2134d3d93d2d47b4 WatchSource:0}: Error finding container 2938c929f268dcee1294b4a934544cb8528f8a09f9e60aec2134d3d93d2d47b4: Status 404 returned error can't find the container with id 2938c929f268dcee1294b4a934544cb8528f8a09f9e60aec2134d3d93d2d47b4
May 28 00:14:19.828320 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:19.828264    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-mtcq5" event={"ID":"2d6f4edf-5055-40c8-a687-08f1753d6bc0","Type":"ContainerStarted","Data":"3ea41919682fd4ff3a3ec4a301a144ad6fd28b5dafdd3b9cc25d7c8197cd4747"}
May 28 00:14:19.828676 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:19.828630    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-mtcq5"
May 28 00:14:19.830489 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:19.830211    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nkppb" event={"ID":"200a652c-a7c2-4aee-8767-26801182f407","Type":"ContainerStarted","Data":"2938c929f268dcee1294b4a934544cb8528f8a09f9e60aec2134d3d93d2d47b4"}
May 28 00:14:19.831833 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:19.831808    2560 generic.go:358] "Generic (PLEG): container finished" podID="80337469-73b3-484b-b1f1-ce8944213baf" containerID="942f81159b4c016d12030a78be2652fa38dac2fbbbbd6d5921f468bd30f9baf1" exitCode=255
May 28 00:14:19.831946 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:19.831850    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp" event={"ID":"80337469-73b3-484b-b1f1-ce8944213baf","Type":"ContainerDied","Data":"942f81159b4c016d12030a78be2652fa38dac2fbbbbd6d5921f468bd30f9baf1"}
May 28 00:14:19.840331 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:19.840314    2560 scope.go:117] "RemoveContainer" containerID="942f81159b4c016d12030a78be2652fa38dac2fbbbbd6d5921f468bd30f9baf1"
May 28 00:14:19.841788 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:19.841764    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-mtcq5"
May 28 00:14:19.855769 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:19.855308    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-mtcq5" podStartSLOduration=2.068805708 podStartE2EDuration="18.855292637s" podCreationTimestamp="2026-05-28 00:14:01 +0000 UTC" firstStartedPulling="2026-05-28 00:14:02.155044707 +0000 UTC m=+151.482710806" lastFinishedPulling="2026-05-28 00:14:18.94153164 +0000 UTC m=+168.269197735" observedRunningTime="2026-05-28 00:14:19.854224192 +0000 UTC m=+169.181890302" watchObservedRunningTime="2026-05-28 00:14:19.855292637 +0000 UTC m=+169.182958745"
May 28 00:14:20.039067 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.038863    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rghgw"]
May 28 00:14:20.041102 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.041073    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.043666 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.043642    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 28 00:14:20.044888 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.043837    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 28 00:14:20.044888 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.043991    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 28 00:14:20.044888 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.044125    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 28 00:14:20.044888 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.044268    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9rdvf\""
May 28 00:14:20.097260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.096801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xph77\" (UniqueName: \"kubernetes.io/projected/6819b656-c4e6-44c5-890d-a52c556e714a-kube-api-access-xph77\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.097260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.096854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-wtmp\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.097260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.096879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6819b656-c4e6-44c5-890d-a52c556e714a-sys\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.097260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.096903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6819b656-c4e6-44c5-890d-a52c556e714a-metrics-client-ca\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.097260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.096938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-tls\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.097260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.096973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.097260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.097029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.097260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.097064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6819b656-c4e6-44c5-890d-a52c556e714a-root\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.097260 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.097089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-textfile\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.197851    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-tls\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.197912    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.197987    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.198030    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6819b656-c4e6-44c5-890d-a52c556e714a-root\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.198054    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-textfile\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.198098    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xph77\" (UniqueName: \"kubernetes.io/projected/6819b656-c4e6-44c5-890d-a52c556e714a-kube-api-access-xph77\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.198131    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-wtmp\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.198159    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6819b656-c4e6-44c5-890d-a52c556e714a-sys\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.198182    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6819b656-c4e6-44c5-890d-a52c556e714a-metrics-client-ca\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.198868    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6819b656-c4e6-44c5-890d-a52c556e714a-metrics-client-ca\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.199127    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-textfile\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.199543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.199138    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-wtmp\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.200421 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.199664    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.200421 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.199708    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6819b656-c4e6-44c5-890d-a52c556e714a-sys\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.200421 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.199738    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6819b656-c4e6-44c5-890d-a52c556e714a-root\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.207672 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.207624    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.209756 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.209695    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xph77\" (UniqueName: \"kubernetes.io/projected/6819b656-c4e6-44c5-890d-a52c556e714a-kube-api-access-xph77\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.218010 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.217809    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6819b656-c4e6-44c5-890d-a52c556e714a-node-exporter-tls\") pod \"node-exporter-rghgw\" (UID: \"6819b656-c4e6-44c5-890d-a52c556e714a\") " pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.256412 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.256377    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:14:20.256919 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.256668    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:14:20.262846 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.262820    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nx64p\""
May 28 00:14:20.271919 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.271740    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4sppr"
May 28 00:14:20.355609 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.355506    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rghgw"
May 28 00:14:20.511495 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:14:20.511446    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6819b656_c4e6_44c5_890d_a52c556e714a.slice/crio-05b2f438beff2f13fbf954a8f5276dff151825de3fcac9b84e861b5ed2d1a9d5 WatchSource:0}: Error finding container 05b2f438beff2f13fbf954a8f5276dff151825de3fcac9b84e861b5ed2d1a9d5: Status 404 returned error can't find the container with id 05b2f438beff2f13fbf954a8f5276dff151825de3fcac9b84e861b5ed2d1a9d5
May 28 00:14:20.657543 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.657478    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4sppr"]
May 28 00:14:20.662491 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:14:20.662452    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2bacc2_9142_49d3_a23c_a62ba15e855e.slice/crio-31dfb5596770c4373107e75e68ea3fe38a16336bfcc5b7f3213e43b93b2e055a WatchSource:0}: Error finding container 31dfb5596770c4373107e75e68ea3fe38a16336bfcc5b7f3213e43b93b2e055a: Status 404 returned error can't find the container with id 31dfb5596770c4373107e75e68ea3fe38a16336bfcc5b7f3213e43b93b2e055a
May 28 00:14:20.837676 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.837634    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4sppr" event={"ID":"da2bacc2-9142-49d3-a23c-a62ba15e855e","Type":"ContainerStarted","Data":"31dfb5596770c4373107e75e68ea3fe38a16336bfcc5b7f3213e43b93b2e055a"}
May 28 00:14:20.839483 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.839453    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nkppb" event={"ID":"200a652c-a7c2-4aee-8767-26801182f407","Type":"ContainerStarted","Data":"40a4757132b7c33979b3df5d5c28ba99ed65c44f29412f72ca71dc920101dbb9"}
May 28 00:14:20.839629 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.839490    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nkppb" event={"ID":"200a652c-a7c2-4aee-8767-26801182f407","Type":"ContainerStarted","Data":"3e675e9b6ee7852b0347083519c36617a2c35f41254cd7cc9588b8d33155625e"}
May 28 00:14:20.839849 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.839808    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nkppb"
May 28 00:14:20.841106 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.841076    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rghgw" event={"ID":"6819b656-c4e6-44c5-890d-a52c556e714a","Type":"ContainerStarted","Data":"05b2f438beff2f13fbf954a8f5276dff151825de3fcac9b84e861b5ed2d1a9d5"}
May 28 00:14:20.843000 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.842961    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f94446f44-h8lwp" event={"ID":"80337469-73b3-484b-b1f1-ce8944213baf","Type":"ContainerStarted","Data":"08aece05fc2af6431cc49abe0bb38922595f1ce787054823031a43496d76237a"}
May 28 00:14:20.859355 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:20.859304    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-nkppb" podStartSLOduration=134.304023871 podStartE2EDuration="2m15.859289046s" podCreationTimestamp="2026-05-28 00:12:05 +0000 UTC" firstStartedPulling="2026-05-28 00:14:18.980363434 +0000 UTC m=+168.308029534" lastFinishedPulling="2026-05-28 00:14:20.535628607 +0000 UTC m=+169.863294709" observedRunningTime="2026-05-28 00:14:20.857671557 +0000 UTC m=+170.185337823" watchObservedRunningTime="2026-05-28 00:14:20.859289046 +0000 UTC m=+170.186955154"
May 28 00:14:21.708897 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:21.708695    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:14:22.054813 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:22.054775    2560 patch_prober.go:28] interesting pod/image-registry-577655c79f-9zmnl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 28 00:14:22.054990 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:22.054838    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-577655c79f-9zmnl" podUID="66068cd5-441a-4aac-819d-47ededa2a63f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 28 00:14:23.778635 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:23.778564    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-577655c79f-9zmnl"
May 28 00:14:23.856206 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:23.856162    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4sppr" event={"ID":"da2bacc2-9142-49d3-a23c-a62ba15e855e","Type":"ContainerStarted","Data":"236af587adc3070a38dd7a5d5a18131bd9de6ec469626c68cb79e2c2b14c6e7b"}
May 28 00:14:23.858061 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:23.858031    2560 generic.go:358] "Generic (PLEG): container finished" podID="6819b656-c4e6-44c5-890d-a52c556e714a" containerID="1e8e94baf24dcb3b9b7f888f9dd535e5e11db1ccef4eaced9a7f7587d4dfdcc3" exitCode=0
May 28 00:14:23.858190 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:23.858106    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rghgw" event={"ID":"6819b656-c4e6-44c5-890d-a52c556e714a","Type":"ContainerDied","Data":"1e8e94baf24dcb3b9b7f888f9dd535e5e11db1ccef4eaced9a7f7587d4dfdcc3"}
May 28 00:14:23.878054 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:23.878008    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4sppr" podStartSLOduration=136.200213594 podStartE2EDuration="2m18.877995656s" podCreationTimestamp="2026-05-28 00:12:05 +0000 UTC" firstStartedPulling="2026-05-28 00:14:20.665342712 +0000 UTC m=+169.993008806" lastFinishedPulling="2026-05-28 00:14:23.343124773 +0000 UTC m=+172.670790868" observedRunningTime="2026-05-28 00:14:23.876259265 +0000 UTC m=+173.203925373" watchObservedRunningTime="2026-05-28 00:14:23.877995656 +0000 UTC m=+173.205661765"
May 28 00:14:24.725780 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.725746    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"]
May 28 00:14:24.737869 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.737841    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"]
May 28 00:14:24.738019 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.737972    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"
May 28 00:14:24.741106 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.741079    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 28 00:14:24.741106 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.741101    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-kxjkm\""
May 28 00:14:24.845602 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.845560    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d360f35-f6bc-43d7-a34e-d8175e7b7a52-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-vdchp\" (UID: \"0d360f35-f6bc-43d7-a34e-d8175e7b7a52\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"
May 28 00:14:24.863409 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.863372    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rghgw" event={"ID":"6819b656-c4e6-44c5-890d-a52c556e714a","Type":"ContainerStarted","Data":"82b73fdeaa3f6bb063f1c877553cc589a20f5f6b36668d92e2724f3e25953612"}
May 28 00:14:24.863559 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.863413    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rghgw" event={"ID":"6819b656-c4e6-44c5-890d-a52c556e714a","Type":"ContainerStarted","Data":"e978106818bdd8fb38071c40407a909d39c1cf8940e749ec8b5b92a1429854d9"}
May 28 00:14:24.882166 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.882125    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rghgw" podStartSLOduration=2.37979687 podStartE2EDuration="4.88211235s" podCreationTimestamp="2026-05-28 00:14:20 +0000 UTC" firstStartedPulling="2026-05-28 00:14:20.525699812 +0000 UTC m=+169.853365914" lastFinishedPulling="2026-05-28 00:14:23.028015295 +0000 UTC m=+172.355681394" observedRunningTime="2026-05-28 00:14:24.88121543 +0000 UTC m=+174.208881538" watchObservedRunningTime="2026-05-28 00:14:24.88211235 +0000 UTC m=+174.209778522"
May 28 00:14:24.946484 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:24.946339    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d360f35-f6bc-43d7-a34e-d8175e7b7a52-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-vdchp\" (UID: \"0d360f35-f6bc-43d7-a34e-d8175e7b7a52\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"
May 28 00:14:24.946662 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:14:24.946591    2560 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 28 00:14:24.946662 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:14:24.946659    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d360f35-f6bc-43d7-a34e-d8175e7b7a52-monitoring-plugin-cert podName:0d360f35-f6bc-43d7-a34e-d8175e7b7a52 nodeName:}" failed. No retries permitted until 2026-05-28 00:14:25.446638413 +0000 UTC m=+174.774304500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0d360f35-f6bc-43d7-a34e-d8175e7b7a52-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-vdchp" (UID: "0d360f35-f6bc-43d7-a34e-d8175e7b7a52") : secret "monitoring-plugin-cert" not found
May 28 00:14:25.449280 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:25.449240    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d360f35-f6bc-43d7-a34e-d8175e7b7a52-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-vdchp\" (UID: \"0d360f35-f6bc-43d7-a34e-d8175e7b7a52\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"
May 28 00:14:25.452216 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:25.452186    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d360f35-f6bc-43d7-a34e-d8175e7b7a52-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-vdchp\" (UID: \"0d360f35-f6bc-43d7-a34e-d8175e7b7a52\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"
May 28 00:14:25.650684 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:25.650644    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"
May 28 00:14:25.797030 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:25.796995    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"]
May 28 00:14:25.800410 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:14:25.800378    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d360f35_f6bc_43d7_a34e_d8175e7b7a52.slice/crio-9499d3577891197c8e95a5ae24fe663af6fbd3f1daf53e51115b18fc925fff40 WatchSource:0}: Error finding container 9499d3577891197c8e95a5ae24fe663af6fbd3f1daf53e51115b18fc925fff40: Status 404 returned error can't find the container with id 9499d3577891197c8e95a5ae24fe663af6fbd3f1daf53e51115b18fc925fff40
May 28 00:14:25.867842 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:25.867807    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp" event={"ID":"0d360f35-f6bc-43d7-a34e-d8175e7b7a52","Type":"ContainerStarted","Data":"9499d3577891197c8e95a5ae24fe663af6fbd3f1daf53e51115b18fc925fff40"}
May 28 00:14:26.721552 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:26.721485    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b8b555b99-htvmk" podUID="5c52f692-08d0-417d-bb51-51beb1c85510" containerName="registry" containerID="cri-o://37d80e50a98bdc292f367e7d5b9f2664cdb6169654a22c60e6884e2a30178c49" gracePeriod=30
May 28 00:14:26.874677 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:26.874639    2560 generic.go:358] "Generic (PLEG): container finished" podID="5c52f692-08d0-417d-bb51-51beb1c85510" containerID="37d80e50a98bdc292f367e7d5b9f2664cdb6169654a22c60e6884e2a30178c49" exitCode=0
May 28 00:14:26.875120 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:26.874717    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b8b555b99-htvmk" event={"ID":"5c52f692-08d0-417d-bb51-51beb1c85510","Type":"ContainerDied","Data":"37d80e50a98bdc292f367e7d5b9f2664cdb6169654a22c60e6884e2a30178c49"}
May 28 00:14:27.011390 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.011366    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:14:27.162258 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162225    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c52f692-08d0-417d-bb51-51beb1c85510-ca-trust-extracted\") pod \"5c52f692-08d0-417d-bb51-51beb1c85510\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") "
May 28 00:14:27.162429 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162278    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-trusted-ca\") pod \"5c52f692-08d0-417d-bb51-51beb1c85510\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") "
May 28 00:14:27.162429 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162315    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-installation-pull-secrets\") pod \"5c52f692-08d0-417d-bb51-51beb1c85510\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") "
May 28 00:14:27.162429 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162357    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-bound-sa-token\") pod \"5c52f692-08d0-417d-bb51-51beb1c85510\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") "
May 28 00:14:27.162611 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162474    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-image-registry-private-configuration\") pod \"5c52f692-08d0-417d-bb51-51beb1c85510\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") "
May 28 00:14:27.162611 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162591    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") pod \"5c52f692-08d0-417d-bb51-51beb1c85510\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") "
May 28 00:14:27.162715 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162653    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-registry-certificates\") pod \"5c52f692-08d0-417d-bb51-51beb1c85510\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") "
May 28 00:14:27.162715 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162681    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fw2z\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-kube-api-access-8fw2z\") pod \"5c52f692-08d0-417d-bb51-51beb1c85510\" (UID: \"5c52f692-08d0-417d-bb51-51beb1c85510\") "
May 28 00:14:27.162814 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162781    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5c52f692-08d0-417d-bb51-51beb1c85510" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:14:27.163577 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.162916    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-trusted-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:14:27.163577 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.163290    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5c52f692-08d0-417d-bb51-51beb1c85510" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:14:27.165927 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.165566    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5c52f692-08d0-417d-bb51-51beb1c85510" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:14:27.166712 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.166682    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-kube-api-access-8fw2z" (OuterVolumeSpecName: "kube-api-access-8fw2z") pod "5c52f692-08d0-417d-bb51-51beb1c85510" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510"). InnerVolumeSpecName "kube-api-access-8fw2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:14:27.175271 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.174856    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5c52f692-08d0-417d-bb51-51beb1c85510" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:14:27.175271 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.175009    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5c52f692-08d0-417d-bb51-51beb1c85510" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:14:27.177861 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.177832    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5c52f692-08d0-417d-bb51-51beb1c85510" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:14:27.180643 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.180619    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c52f692-08d0-417d-bb51-51beb1c85510-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5c52f692-08d0-417d-bb51-51beb1c85510" (UID: "5c52f692-08d0-417d-bb51-51beb1c85510"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:14:27.263651 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.263577    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-installation-pull-secrets\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:14:27.263651 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.263610    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-bound-sa-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:14:27.263651 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.263627    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c52f692-08d0-417d-bb51-51beb1c85510-image-registry-private-configuration\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:14:27.263651 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.263644    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-registry-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:14:27.263973 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.263658    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c52f692-08d0-417d-bb51-51beb1c85510-registry-certificates\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:14:27.263973 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.263672    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fw2z\" (UniqueName: \"kubernetes.io/projected/5c52f692-08d0-417d-bb51-51beb1c85510-kube-api-access-8fw2z\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:14:27.263973 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.263687    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c52f692-08d0-417d-bb51-51beb1c85510-ca-trust-extracted\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:14:27.879051 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.879005    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b8b555b99-htvmk" event={"ID":"5c52f692-08d0-417d-bb51-51beb1c85510","Type":"ContainerDied","Data":"94e05978519be7caa2750cf9ecefe185c5c1416216447baf2dfbbdab8e64a4cd"}
May 28 00:14:27.879510 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.879074    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b8b555b99-htvmk"
May 28 00:14:27.879510 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.879074    2560 scope.go:117] "RemoveContainer" containerID="37d80e50a98bdc292f367e7d5b9f2664cdb6169654a22c60e6884e2a30178c49"
May 28 00:14:27.894045 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.894019    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b8b555b99-htvmk"]
May 28 00:14:27.897615 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:27.897587    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b8b555b99-htvmk"]
May 28 00:14:28.883452 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:28.883421    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp" event={"ID":"0d360f35-f6bc-43d7-a34e-d8175e7b7a52","Type":"ContainerStarted","Data":"01a6f7f30565d9c9ef25abd3ea4ef2cda4668a423b02aa15459b1d2deda79ddf"}
May 28 00:14:28.883903 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:28.883636    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"
May 28 00:14:28.888245 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:28.888224    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp"
May 28 00:14:28.899740 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:28.899677    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-vdchp" podStartSLOduration=2.634060846 podStartE2EDuration="4.899665007s" podCreationTimestamp="2026-05-28 00:14:24 +0000 UTC" firstStartedPulling="2026-05-28 00:14:25.802659388 +0000 UTC m=+175.130325474" lastFinishedPulling="2026-05-28 00:14:28.068263536 +0000 UTC m=+177.395929635" observedRunningTime="2026-05-28 00:14:28.898890613 +0000 UTC m=+178.226556722" watchObservedRunningTime="2026-05-28 00:14:28.899665007 +0000 UTC m=+178.227331118"
May 28 00:14:29.260173 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:29.260103    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c52f692-08d0-417d-bb51-51beb1c85510" path="/var/lib/kubelet/pods/5c52f692-08d0-417d-bb51-51beb1c85510/volumes"
May 28 00:14:30.849684 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:30.849651    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nkppb"
May 28 00:14:53.952290 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:53.952205    2560 generic.go:358] "Generic (PLEG): container finished" podID="abfdf3bc-d254-45a0-b65c-5a67c4490eb8" containerID="3374430f39b0e04556d5f3a9f79f866599d2c1cb196567b4cb6bd0b044dcf5a9" exitCode=0
May 28 00:14:53.952793 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:53.952284    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw" event={"ID":"abfdf3bc-d254-45a0-b65c-5a67c4490eb8","Type":"ContainerDied","Data":"3374430f39b0e04556d5f3a9f79f866599d2c1cb196567b4cb6bd0b044dcf5a9"}
May 28 00:14:53.952793 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:53.952656    2560 scope.go:117] "RemoveContainer" containerID="3374430f39b0e04556d5f3a9f79f866599d2c1cb196567b4cb6bd0b044dcf5a9"
May 28 00:14:54.956763 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:14:54.956730    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-m7mzw" event={"ID":"abfdf3bc-d254-45a0-b65c-5a67c4490eb8","Type":"ContainerStarted","Data":"f2bbc09b4fc710574e5f13a3cb4c3a5a84343d1fbb01e57d603a04c9a51f96e9"}
May 28 00:15:43.012505 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:15:43.012421    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:15:43.014845 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:15:43.014820    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0d3ce5a-3b25-4284-8946-bbb3e5820ce6-metrics-certs\") pod \"network-metrics-daemon-2dnl7\" (UID: \"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6\") " pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:15:43.059378 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:15:43.059348    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fp9tk\""
May 28 00:15:43.066995 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:15:43.066973    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2dnl7"
May 28 00:15:43.200651 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:15:43.200571    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2dnl7"]
May 28 00:15:43.203300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:15:43.203271    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d3ce5a_3b25_4284_8946_bbb3e5820ce6.slice/crio-1d0eda1957ac13cb66d9a8a6951d74daded4c9eb96f9cb92cbac4c3dc85675e2 WatchSource:0}: Error finding container 1d0eda1957ac13cb66d9a8a6951d74daded4c9eb96f9cb92cbac4c3dc85675e2: Status 404 returned error can't find the container with id 1d0eda1957ac13cb66d9a8a6951d74daded4c9eb96f9cb92cbac4c3dc85675e2
May 28 00:15:44.089338 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:15:44.089306    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2dnl7" event={"ID":"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6","Type":"ContainerStarted","Data":"1d0eda1957ac13cb66d9a8a6951d74daded4c9eb96f9cb92cbac4c3dc85675e2"}
May 28 00:15:45.093376 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:15:45.093342    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2dnl7" event={"ID":"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6","Type":"ContainerStarted","Data":"ccf0fe71d71fcee3f26770898e9b21840f65fbe2d94f88a20bdb4e1c2648ca80"}
May 28 00:15:45.093376 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:15:45.093380    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2dnl7" event={"ID":"c0d3ce5a-3b25-4284-8946-bbb3e5820ce6","Type":"ContainerStarted","Data":"8aca70b9c67a37a882a3678b822bac08c7e495fb9add37c568600436be255b5e"}
May 28 00:15:45.113074 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:15:45.113015    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2dnl7" podStartSLOduration=253.300870459 podStartE2EDuration="4m14.112997621s" podCreationTimestamp="2026-05-28 00:11:31 +0000 UTC" firstStartedPulling="2026-05-28 00:15:43.205010542 +0000 UTC m=+252.532676629" lastFinishedPulling="2026-05-28 00:15:44.017137696 +0000 UTC m=+253.344803791" observedRunningTime="2026-05-28 00:15:45.111877494 +0000 UTC m=+254.439543603" watchObservedRunningTime="2026-05-28 00:15:45.112997621 +0000 UTC m=+254.440663730"
May 28 00:16:20.766603 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.766566    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-87tk8"]
May 28 00:16:20.767115 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.766802    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c52f692-08d0-417d-bb51-51beb1c85510" containerName="registry"
May 28 00:16:20.767115 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.766813    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c52f692-08d0-417d-bb51-51beb1c85510" containerName="registry"
May 28 00:16:20.767115 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.766858    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c52f692-08d0-417d-bb51-51beb1c85510" containerName="registry"
May 28 00:16:20.768604 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.768587    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:20.771632 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.771611    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 28 00:16:20.777918 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.777898    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-87tk8"]
May 28 00:16:20.881524 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.881494    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112-kubelet-config\") pod \"global-pull-secret-syncer-87tk8\" (UID: \"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112\") " pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:20.881644 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.881551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112-dbus\") pod \"global-pull-secret-syncer-87tk8\" (UID: \"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112\") " pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:20.881644 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.881575    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112-original-pull-secret\") pod \"global-pull-secret-syncer-87tk8\" (UID: \"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112\") " pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:20.982410 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.982386    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112-kubelet-config\") pod \"global-pull-secret-syncer-87tk8\" (UID: \"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112\") " pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:20.982496 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.982421    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112-dbus\") pod \"global-pull-secret-syncer-87tk8\" (UID: \"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112\") " pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:20.982496 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.982455    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112-original-pull-secret\") pod \"global-pull-secret-syncer-87tk8\" (UID: \"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112\") " pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:20.982601 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.982503    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112-kubelet-config\") pod \"global-pull-secret-syncer-87tk8\" (UID: \"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112\") " pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:20.982645 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.982601    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112-dbus\") pod \"global-pull-secret-syncer-87tk8\" (UID: \"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112\") " pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:20.984786 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:20.984766    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112-original-pull-secret\") pod \"global-pull-secret-syncer-87tk8\" (UID: \"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112\") " pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:21.077339 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:21.077296    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-87tk8"
May 28 00:16:21.196884 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:21.196827    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-87tk8"]
May 28 00:16:21.200129 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:16:21.200101    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d0ed4b_1a8c_4e6f_80ea_a56a3a18d112.slice/crio-7f0f3c54db8727be0dd3aac1ae08b426bc605310d6ab0b0cdcc76609edbf0306 WatchSource:0}: Error finding container 7f0f3c54db8727be0dd3aac1ae08b426bc605310d6ab0b0cdcc76609edbf0306: Status 404 returned error can't find the container with id 7f0f3c54db8727be0dd3aac1ae08b426bc605310d6ab0b0cdcc76609edbf0306
May 28 00:16:22.190861 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:22.190829    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-87tk8" event={"ID":"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112","Type":"ContainerStarted","Data":"7f0f3c54db8727be0dd3aac1ae08b426bc605310d6ab0b0cdcc76609edbf0306"}
May 28 00:16:26.202533 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:26.202490    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-87tk8" event={"ID":"57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112","Type":"ContainerStarted","Data":"0892eb9fd69aa8edfa3b0f770cc0437d675975282536388dcbdeb4ba48745e16"}
May 28 00:16:26.219474 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:26.219425    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-87tk8" podStartSLOduration=2.293111195 podStartE2EDuration="6.219412505s" podCreationTimestamp="2026-05-28 00:16:20 +0000 UTC" firstStartedPulling="2026-05-28 00:16:21.201599848 +0000 UTC m=+290.529265934" lastFinishedPulling="2026-05-28 00:16:25.127901156 +0000 UTC m=+294.455567244" observedRunningTime="2026-05-28 00:16:26.217887071 +0000 UTC m=+295.545553179" watchObservedRunningTime="2026-05-28 00:16:26.219412505 +0000 UTC m=+295.547078613"
May 28 00:16:31.151119 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:16:31.151091    2560 kubelet.go:1628] "Image garbage collection succeeded"
May 28 00:17:58.789233 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:58.789199    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"]
May 28 00:17:58.791257 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:58.791238    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"
May 28 00:17:58.793003 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:58.792979    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 28 00:17:58.793124 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:58.793106    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 28 00:17:58.793392 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:58.793377    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-gnlrp\""
May 28 00:17:58.800623 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:58.800597    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"]
May 28 00:17:58.956774 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:58.956728    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp2hl\" (UniqueName: \"kubernetes.io/projected/23235329-ac8d-48b5-a11e-608bc4e9f125-kube-api-access-dp2hl\") pod \"feast-operator-controller-manager-5f9b99fd4f-2hlhx\" (UID: \"23235329-ac8d-48b5-a11e-608bc4e9f125\") " pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"
May 28 00:17:59.057648 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:59.057548    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dp2hl\" (UniqueName: \"kubernetes.io/projected/23235329-ac8d-48b5-a11e-608bc4e9f125-kube-api-access-dp2hl\") pod \"feast-operator-controller-manager-5f9b99fd4f-2hlhx\" (UID: \"23235329-ac8d-48b5-a11e-608bc4e9f125\") " pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"
May 28 00:17:59.066302 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:59.066278    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp2hl\" (UniqueName: \"kubernetes.io/projected/23235329-ac8d-48b5-a11e-608bc4e9f125-kube-api-access-dp2hl\") pod \"feast-operator-controller-manager-5f9b99fd4f-2hlhx\" (UID: \"23235329-ac8d-48b5-a11e-608bc4e9f125\") " pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"
May 28 00:17:59.102263 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:59.102220    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"
May 28 00:17:59.224609 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:59.224489    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"]
May 28 00:17:59.227113 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:17:59.227085    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23235329_ac8d_48b5_a11e_608bc4e9f125.slice/crio-b0f8e8c459c1ccbda3414ddc944eaf3fa48aa3df4986cb2cd793633cab2fe495 WatchSource:0}: Error finding container b0f8e8c459c1ccbda3414ddc944eaf3fa48aa3df4986cb2cd793633cab2fe495: Status 404 returned error can't find the container with id b0f8e8c459c1ccbda3414ddc944eaf3fa48aa3df4986cb2cd793633cab2fe495
May 28 00:17:59.228972 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:59.228955    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:17:59.444148 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:17:59.444061    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx" event={"ID":"23235329-ac8d-48b5-a11e-608bc4e9f125","Type":"ContainerStarted","Data":"b0f8e8c459c1ccbda3414ddc944eaf3fa48aa3df4986cb2cd793633cab2fe495"}
May 28 00:18:03.456381 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:18:03.456345    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx" event={"ID":"23235329-ac8d-48b5-a11e-608bc4e9f125","Type":"ContainerStarted","Data":"9dc21ec27d3ffa1fabaa3a44be237bfc0742e4c9c504642fe88784c44e292388"}
May 28 00:18:03.456871 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:18:03.456433    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"
May 28 00:18:03.471363 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:18:03.471317    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx" podStartSLOduration=2.231316956 podStartE2EDuration="5.471305105s" podCreationTimestamp="2026-05-28 00:17:58 +0000 UTC" firstStartedPulling="2026-05-28 00:17:59.229106166 +0000 UTC m=+388.556772253" lastFinishedPulling="2026-05-28 00:18:02.469094313 +0000 UTC m=+391.796760402" observedRunningTime="2026-05-28 00:18:03.469723686 +0000 UTC m=+392.797389805" watchObservedRunningTime="2026-05-28 00:18:03.471305105 +0000 UTC m=+392.798971214"
May 28 00:18:14.464002 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:18:14.463967    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"
May 28 00:24:36.809363 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:36.809328    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-test-apply-b6c52"]
May 28 00:24:36.812360 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:36.812344    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-b6c52"
May 28 00:24:36.814254 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:36.814227    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 28 00:24:36.814365 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:36.814346    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-lt26w\""
May 28 00:24:36.814411 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:36.814355    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 00:24:36.814695 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:36.814682    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 00:24:36.817984 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:36.817963    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-b6c52"]
May 28 00:24:36.934436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:36.934405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz85g\" (UniqueName: \"kubernetes.io/projected/9b71c910-b8aa-4c9f-81bf-0357bd0026d3-kube-api-access-xz85g\") pod \"feast-test-apply-b6c52\" (UID: \"9b71c910-b8aa-4c9f-81bf-0357bd0026d3\") " pod="test-ns-feast/feast-test-apply-b6c52"
May 28 00:24:37.034760 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:37.034727    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xz85g\" (UniqueName: \"kubernetes.io/projected/9b71c910-b8aa-4c9f-81bf-0357bd0026d3-kube-api-access-xz85g\") pod \"feast-test-apply-b6c52\" (UID: \"9b71c910-b8aa-4c9f-81bf-0357bd0026d3\") " pod="test-ns-feast/feast-test-apply-b6c52"
May 28 00:24:37.042840 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:37.042817    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz85g\" (UniqueName: \"kubernetes.io/projected/9b71c910-b8aa-4c9f-81bf-0357bd0026d3-kube-api-access-xz85g\") pod \"feast-test-apply-b6c52\" (UID: \"9b71c910-b8aa-4c9f-81bf-0357bd0026d3\") " pod="test-ns-feast/feast-test-apply-b6c52"
May 28 00:24:37.122390 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:37.122332    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-b6c52"
May 28 00:24:37.244048 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:37.244023    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-b6c52"]
May 28 00:24:37.246703 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:24:37.246676    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b71c910_b8aa_4c9f_81bf_0357bd0026d3.slice/crio-6f418f8f8776e296f65ad79b5d89093a5fee8b66a9b23e95d469d68cef1b0dc2 WatchSource:0}: Error finding container 6f418f8f8776e296f65ad79b5d89093a5fee8b66a9b23e95d469d68cef1b0dc2: Status 404 returned error can't find the container with id 6f418f8f8776e296f65ad79b5d89093a5fee8b66a9b23e95d469d68cef1b0dc2
May 28 00:24:37.248423 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:37.248406    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:24:37.463285 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:37.463198    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-b6c52" event={"ID":"9b71c910-b8aa-4c9f-81bf-0357bd0026d3","Type":"ContainerStarted","Data":"6f418f8f8776e296f65ad79b5d89093a5fee8b66a9b23e95d469d68cef1b0dc2"}
May 28 00:24:43.484578 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:43.484475    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-b6c52" event={"ID":"9b71c910-b8aa-4c9f-81bf-0357bd0026d3","Type":"ContainerStarted","Data":"5da6d32251685fdd8b9543910b900b2c21b908412521ed6cb4a7665fffc5b018"}
May 28 00:24:49.500396 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:49.500359    2560 generic.go:358] "Generic (PLEG): container finished" podID="9b71c910-b8aa-4c9f-81bf-0357bd0026d3" containerID="5da6d32251685fdd8b9543910b900b2c21b908412521ed6cb4a7665fffc5b018" exitCode=0
May 28 00:24:49.500741 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:49.500431    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-b6c52" event={"ID":"9b71c910-b8aa-4c9f-81bf-0357bd0026d3","Type":"ContainerDied","Data":"5da6d32251685fdd8b9543910b900b2c21b908412521ed6cb4a7665fffc5b018"}
May 28 00:24:50.504879 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:50.504845    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-b6c52" event={"ID":"9b71c910-b8aa-4c9f-81bf-0357bd0026d3","Type":"ContainerStarted","Data":"c743ccf05acc87c7bccced955d0e0f78a5c943fc1c941e64ebbf9eed71f2d90d"}
May 28 00:24:50.522635 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:50.522580    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-test-apply-b6c52" podStartSLOduration=8.559339115 podStartE2EDuration="14.522562448s" podCreationTimestamp="2026-05-28 00:24:36 +0000 UTC" firstStartedPulling="2026-05-28 00:24:37.248561465 +0000 UTC m=+786.576227551" lastFinishedPulling="2026-05-28 00:24:43.211784798 +0000 UTC m=+792.539450884" observedRunningTime="2026-05-28 00:24:50.521269008 +0000 UTC m=+799.848935118" watchObservedRunningTime="2026-05-28 00:24:50.522562448 +0000 UTC m=+799.850228557"
May 28 00:24:59.527720 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:59.527680    2560 generic.go:358] "Generic (PLEG): container finished" podID="9b71c910-b8aa-4c9f-81bf-0357bd0026d3" containerID="c743ccf05acc87c7bccced955d0e0f78a5c943fc1c941e64ebbf9eed71f2d90d" exitCode=0
May 28 00:24:59.528113 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:24:59.527756    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-b6c52" event={"ID":"9b71c910-b8aa-4c9f-81bf-0357bd0026d3","Type":"ContainerDied","Data":"c743ccf05acc87c7bccced955d0e0f78a5c943fc1c941e64ebbf9eed71f2d90d"}
May 28 00:25:00.660946 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:00.660926    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-b6c52"
May 28 00:25:00.700792 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:00.700761    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz85g\" (UniqueName: \"kubernetes.io/projected/9b71c910-b8aa-4c9f-81bf-0357bd0026d3-kube-api-access-xz85g\") pod \"9b71c910-b8aa-4c9f-81bf-0357bd0026d3\" (UID: \"9b71c910-b8aa-4c9f-81bf-0357bd0026d3\") "
May 28 00:25:00.702924 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:00.702901    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b71c910-b8aa-4c9f-81bf-0357bd0026d3-kube-api-access-xz85g" (OuterVolumeSpecName: "kube-api-access-xz85g") pod "9b71c910-b8aa-4c9f-81bf-0357bd0026d3" (UID: "9b71c910-b8aa-4c9f-81bf-0357bd0026d3"). InnerVolumeSpecName "kube-api-access-xz85g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:25:00.801569 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:00.801548    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz85g\" (UniqueName: \"kubernetes.io/projected/9b71c910-b8aa-4c9f-81bf-0357bd0026d3-kube-api-access-xz85g\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:25:01.534257 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:01.534236    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-b6c52"
May 28 00:25:01.534386 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:01.534250    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-b6c52" event={"ID":"9b71c910-b8aa-4c9f-81bf-0357bd0026d3","Type":"ContainerDied","Data":"6f418f8f8776e296f65ad79b5d89093a5fee8b66a9b23e95d469d68cef1b0dc2"}
May 28 00:25:01.534386 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:01.534279    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f418f8f8776e296f65ad79b5d89093a5fee8b66a9b23e95d469d68cef1b0dc2"
May 28 00:25:01.714940 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:01.714916    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-b6c52_9b71c910-b8aa-4c9f-81bf-0357bd0026d3/feast-0/0.log"
May 28 00:25:01.721667 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:01.721645    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-b6c52_9b71c910-b8aa-4c9f-81bf-0357bd0026d3/feast-1/0.log"
May 28 00:25:37.845674 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:37.845641    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-test-apply-b6c52"]
May 28 00:25:37.857759 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:37.857732    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-test-apply-b6c52"]
May 28 00:25:39.260124 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:25:39.260093    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b71c910-b8aa-4c9f-81bf-0357bd0026d3" path="/var/lib/kubelet/pods/9b71c910-b8aa-4c9f-81bf-0357bd0026d3/volumes"
May 28 00:30:20.143593 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:20.143552    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"]
May 28 00:30:20.144180 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:20.143818    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx" podUID="23235329-ac8d-48b5-a11e-608bc4e9f125" containerName="manager" containerID="cri-o://9dc21ec27d3ffa1fabaa3a44be237bfc0742e4c9c504642fe88784c44e292388" gracePeriod=10
May 28 00:30:20.370325 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:20.370292    2560 generic.go:358] "Generic (PLEG): container finished" podID="23235329-ac8d-48b5-a11e-608bc4e9f125" containerID="9dc21ec27d3ffa1fabaa3a44be237bfc0742e4c9c504642fe88784c44e292388" exitCode=0
May 28 00:30:20.370492 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:20.370333    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx" event={"ID":"23235329-ac8d-48b5-a11e-608bc4e9f125","Type":"ContainerDied","Data":"9dc21ec27d3ffa1fabaa3a44be237bfc0742e4c9c504642fe88784c44e292388"}
May 28 00:30:20.404184 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:20.404125    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"
May 28 00:30:20.441628 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:20.441591    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp2hl\" (UniqueName: \"kubernetes.io/projected/23235329-ac8d-48b5-a11e-608bc4e9f125-kube-api-access-dp2hl\") pod \"23235329-ac8d-48b5-a11e-608bc4e9f125\" (UID: \"23235329-ac8d-48b5-a11e-608bc4e9f125\") "
May 28 00:30:20.443715 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:20.443687    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23235329-ac8d-48b5-a11e-608bc4e9f125-kube-api-access-dp2hl" (OuterVolumeSpecName: "kube-api-access-dp2hl") pod "23235329-ac8d-48b5-a11e-608bc4e9f125" (UID: "23235329-ac8d-48b5-a11e-608bc4e9f125"). InnerVolumeSpecName "kube-api-access-dp2hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:30:20.542707 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:20.542674    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp2hl\" (UniqueName: \"kubernetes.io/projected/23235329-ac8d-48b5-a11e-608bc4e9f125-kube-api-access-dp2hl\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:30:21.373796 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:21.373761    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx" event={"ID":"23235329-ac8d-48b5-a11e-608bc4e9f125","Type":"ContainerDied","Data":"b0f8e8c459c1ccbda3414ddc944eaf3fa48aa3df4986cb2cd793633cab2fe495"}
May 28 00:30:21.374196 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:21.373806    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"
May 28 00:30:21.374196 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:21.373810    2560 scope.go:117] "RemoveContainer" containerID="9dc21ec27d3ffa1fabaa3a44be237bfc0742e4c9c504642fe88784c44e292388"
May 28 00:30:21.388710 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:21.388686    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"]
May 28 00:30:21.392469 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:21.392439    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f9b99fd4f-2hlhx"]
May 28 00:30:23.264171 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:23.264140    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23235329-ac8d-48b5-a11e-608bc4e9f125" path="/var/lib/kubelet/pods/23235329-ac8d-48b5-a11e-608bc4e9f125/volumes"
May 28 00:30:31.582736 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.582706    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"]
May 28 00:30:31.583118 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.582938    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23235329-ac8d-48b5-a11e-608bc4e9f125" containerName="manager"
May 28 00:30:31.583118 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.582948    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="23235329-ac8d-48b5-a11e-608bc4e9f125" containerName="manager"
May 28 00:30:31.583118 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.582955    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b71c910-b8aa-4c9f-81bf-0357bd0026d3" containerName="feast-0"
May 28 00:30:31.583118 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.582961    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b71c910-b8aa-4c9f-81bf-0357bd0026d3" containerName="feast-0"
May 28 00:30:31.583118 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.582971    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b71c910-b8aa-4c9f-81bf-0357bd0026d3" containerName="feast-1"
May 28 00:30:31.583118 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.582976    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b71c910-b8aa-4c9f-81bf-0357bd0026d3" containerName="feast-1"
May 28 00:30:31.583118 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.583021    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="9b71c910-b8aa-4c9f-81bf-0357bd0026d3" containerName="feast-1"
May 28 00:30:31.583118 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.583030    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="23235329-ac8d-48b5-a11e-608bc4e9f125" containerName="manager"
May 28 00:30:31.587008 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.586992    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"
May 28 00:30:31.590674 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.590649    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 28 00:30:31.591066 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.591050    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-2xpq9\""
May 28 00:30:31.591197 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.591178    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 28 00:30:31.610622 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.610597    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"]
May 28 00:30:31.622320 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.622300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwhz\" (UniqueName: \"kubernetes.io/projected/dbde35c5-a5df-43a2-a7ba-1cd28e0443b8-kube-api-access-xqwhz\") pod \"feast-operator-controller-manager-cf99db75-h54dx\" (UID: \"dbde35c5-a5df-43a2-a7ba-1cd28e0443b8\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"
May 28 00:30:31.722841 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.722802    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwhz\" (UniqueName: \"kubernetes.io/projected/dbde35c5-a5df-43a2-a7ba-1cd28e0443b8-kube-api-access-xqwhz\") pod \"feast-operator-controller-manager-cf99db75-h54dx\" (UID: \"dbde35c5-a5df-43a2-a7ba-1cd28e0443b8\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"
May 28 00:30:31.735217 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.735198    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwhz\" (UniqueName: \"kubernetes.io/projected/dbde35c5-a5df-43a2-a7ba-1cd28e0443b8-kube-api-access-xqwhz\") pod \"feast-operator-controller-manager-cf99db75-h54dx\" (UID: \"dbde35c5-a5df-43a2-a7ba-1cd28e0443b8\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"
May 28 00:30:31.896252 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:31.896164    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"
May 28 00:30:32.027307 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:32.027284    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"]
May 28 00:30:32.029175 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:30:32.029147    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbde35c5_a5df_43a2_a7ba_1cd28e0443b8.slice/crio-15fde7c67f19c690e5dc030afe38874123b752554a437e8308eb42b7b31331c6 WatchSource:0}: Error finding container 15fde7c67f19c690e5dc030afe38874123b752554a437e8308eb42b7b31331c6: Status 404 returned error can't find the container with id 15fde7c67f19c690e5dc030afe38874123b752554a437e8308eb42b7b31331c6
May 28 00:30:32.031114 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:32.031097    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:30:32.403955 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:32.403918    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx" event={"ID":"dbde35c5-a5df-43a2-a7ba-1cd28e0443b8","Type":"ContainerStarted","Data":"15fde7c67f19c690e5dc030afe38874123b752554a437e8308eb42b7b31331c6"}
May 28 00:30:34.410905 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:34.410867    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx" event={"ID":"dbde35c5-a5df-43a2-a7ba-1cd28e0443b8","Type":"ContainerStarted","Data":"4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff"}
May 28 00:30:34.411356 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:34.410962    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"
May 28 00:30:34.438145 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:34.438051    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx" podStartSLOduration=1.225973278 podStartE2EDuration="3.438033671s" podCreationTimestamp="2026-05-28 00:30:31 +0000 UTC" firstStartedPulling="2026-05-28 00:30:32.031252385 +0000 UTC m=+1141.358918475" lastFinishedPulling="2026-05-28 00:30:34.243312782 +0000 UTC m=+1143.570978868" observedRunningTime="2026-05-28 00:30:34.437869099 +0000 UTC m=+1143.765535207" watchObservedRunningTime="2026-05-28 00:30:34.438033671 +0000 UTC m=+1143.765699781"
May 28 00:30:45.415911 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:45.415830    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"
May 28 00:30:46.385077 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.385043    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"]
May 28 00:30:46.388471 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.388452    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.392886 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.392863    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 00:30:46.393020 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.392963    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 00:30:46.393020 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.392983    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 00:30:46.393128 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.393016    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-wtdqf\""
May 28 00:30:46.393346 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.393334    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 00:30:46.393409 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.393391    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 00:30:46.397080 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.397062    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 00:30:46.429863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.429838    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"]
May 28 00:30:46.532929 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.532893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.533089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.532939    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.533089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.532969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.533089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.533030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.533089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.533060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8bt\" (UniqueName: \"kubernetes.io/projected/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-kube-api-access-qf8bt\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.533218 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.533089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.633590 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.633557    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.633744 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.633598    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.633744 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.633621    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8bt\" (UniqueName: \"kubernetes.io/projected/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-kube-api-access-qf8bt\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.633744 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.633640    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.633744 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.633683    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.633744 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.633712    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.633744 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:30:46.633731    2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 00:30:46.634056 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:30:46.633807    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-registry-tls podName:62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d nodeName:}" failed. No retries permitted until 2026-05-28 00:30:47.133787263 +0000 UTC m=+1156.461453348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-wds2f" (UID: "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 00:30:46.634056 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:30:46.633813    2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 28 00:30:46.634056 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:30:46.633877    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-offline-tls podName:62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d nodeName:}" failed. No retries permitted until 2026-05-28 00:30:47.133860574 +0000 UTC m=+1156.461526664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-wds2f" (UID: "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d") : secret "feast-simple-feast-setup-offline-tls" not found
May 28 00:30:46.634056 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.634027    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.636271 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.636227    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.636629 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.636602    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:46.647300 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:46.647275    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8bt\" (UniqueName: \"kubernetes.io/projected/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-kube-api-access-qf8bt\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:47.137510 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:47.137471    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:47.137686 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:47.137535    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:47.140098 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:47.140069    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:47.140206 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:47.140110    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-wds2f\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:47.297872 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:47.297837    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:30:47.443390 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:47.443257    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"]
May 28 00:30:47.446124 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:30:47.446096    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62bf816e_ee87_4c31_a2ed_5e73ad2f9d9d.slice/crio-dbfab8c8690e661a63091d55d0523333641e07c2bdfe575d9da5ec001b932f10 WatchSource:0}: Error finding container dbfab8c8690e661a63091d55d0523333641e07c2bdfe575d9da5ec001b932f10: Status 404 returned error can't find the container with id dbfab8c8690e661a63091d55d0523333641e07c2bdfe575d9da5ec001b932f10
May 28 00:30:48.463844 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:30:48.463776    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerStarted","Data":"dbfab8c8690e661a63091d55d0523333641e07c2bdfe575d9da5ec001b932f10"}
May 28 00:31:04.512229 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:04.512193    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerStarted","Data":"d918ef00ccaa1562ffb62a355cd60683e38836acd82039f885710d1f2bcd01c2"}
May 28 00:31:08.524998 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:08.524961    2560 generic.go:358] "Generic (PLEG): container finished" podID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerID="d918ef00ccaa1562ffb62a355cd60683e38836acd82039f885710d1f2bcd01c2" exitCode=0
May 28 00:31:08.525318 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:08.525033    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerDied","Data":"d918ef00ccaa1562ffb62a355cd60683e38836acd82039f885710d1f2bcd01c2"}
May 28 00:31:09.529933 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:09.529898    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerStarted","Data":"6cd84ac2986c76cbeab9205e62be2dfa7f76e19a7c645f309381e31d7e59b531"}
May 28 00:31:14.544060 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:14.544025    2560 generic.go:358] "Generic (PLEG): container finished" podID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerID="6cd84ac2986c76cbeab9205e62be2dfa7f76e19a7c645f309381e31d7e59b531" exitCode=0
May 28 00:31:14.544458 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:14.544072    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerDied","Data":"6cd84ac2986c76cbeab9205e62be2dfa7f76e19a7c645f309381e31d7e59b531"}
May 28 00:31:15.551599 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:15.551549    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerStarted","Data":"688957703c62f2c61593ba5cce8054db17fbe9a9891aced251a020a58e73ecfb"}
May 28 00:31:15.552067 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:15.551611    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerStarted","Data":"949c1ac98ab049376bd27e7cca37fe8d02ecc1e00925d85e401b14ccb218794e"}
May 28 00:31:15.552067 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:15.551627    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerStarted","Data":"00825903c87c970911673e0aaf5ca9729fa305ca9135eeece75a5a27c2dfc02f"}
May 28 00:31:15.552067 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:15.551639    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerStarted","Data":"551a16e85642df153b892ed3909952daa28718dba78606ac3a9d6ef001f4b531"}
May 28 00:31:15.579341 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:15.577613    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podStartSLOduration=13.10012411 podStartE2EDuration="29.57759703s" podCreationTimestamp="2026-05-28 00:30:46 +0000 UTC" firstStartedPulling="2026-05-28 00:30:47.447678249 +0000 UTC m=+1156.775344338" lastFinishedPulling="2026-05-28 00:31:03.925151171 +0000 UTC m=+1173.252817258" observedRunningTime="2026-05-28 00:31:15.576493637 +0000 UTC m=+1184.904159771" watchObservedRunningTime="2026-05-28 00:31:15.57759703 +0000 UTC m=+1184.905263139"
May 28 00:31:17.298284 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.298031    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:17.298284 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.298089    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:17.298284 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.298104    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:17.298284 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.298117    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:17.298284 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.298128    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:17.298284 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.298140    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:17.299039 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.298617    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:17.299095 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.299042    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:17.300434 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.300390    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.21:8816: connect: connection refused"
May 28 00:31:17.300434 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.300419    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused"
May 28 00:31:17.300732 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.300446    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused"
May 28 00:31:17.300732 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:17.300495    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused"
May 28 00:31:20.299341 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:20.299310    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:20.299852 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:20.299421    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:20.299852 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:20.299665    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:20.300396 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:20.300374    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:20.300529 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:20.300485    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:20.301677 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:20.301659    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:20.304181 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:20.304165    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:20.570294 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:20.570217    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:31:31.209710 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:31.209682    2560 scope.go:117] "RemoveContainer" containerID="c743ccf05acc87c7bccced955d0e0f78a5c943fc1c941e64ebbf9eed71f2d90d"
May 28 00:31:31.216894 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:31:31.216879    2560 scope.go:117] "RemoveContainer" containerID="5da6d32251685fdd8b9543910b900b2c21b908412521ed6cb4a7665fffc5b018"
May 28 00:32:23.504685 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.504452    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"]
May 28 00:32:23.505223 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.504954    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="registry" containerID="cri-o://551a16e85642df153b892ed3909952daa28718dba78606ac3a9d6ef001f4b531" gracePeriod=30
May 28 00:32:23.505363 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.505337    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="ui" containerID="cri-o://688957703c62f2c61593ba5cce8054db17fbe9a9891aced251a020a58e73ecfb" gracePeriod=30
May 28 00:32:23.505585 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.505560    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="offline" containerID="cri-o://949c1ac98ab049376bd27e7cca37fe8d02ecc1e00925d85e401b14ccb218794e" gracePeriod=30
May 28 00:32:23.505788 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.505762    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="online" containerID="cri-o://00825903c87c970911673e0aaf5ca9729fa305ca9135eeece75a5a27c2dfc02f" gracePeriod=30
May 28 00:32:23.816757 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.816721    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"]
May 28 00:32:23.819662 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.819638    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.821779 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.821758    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-l4mwc\""
May 28 00:32:23.835175 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.835149    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"]
May 28 00:32:23.856699 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.856670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.856870 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.856723    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.856870 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.856753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k24j\" (UniqueName: \"kubernetes.io/projected/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-kube-api-access-9k24j\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.856870 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.856791    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.856870 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.856821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.856870 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.856850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.957277 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.957239    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.957463 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.957287    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.957463 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.957319    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9k24j\" (UniqueName: \"kubernetes.io/projected/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-kube-api-access-9k24j\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.957463 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.957355    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.957463 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.957384    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.957463 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:32:23.957389    2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 00:32:23.957463 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.957414    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.957463 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:32:23.957457    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-registry-tls podName:d066b84c-9cf1-4386-821a-9b0c9d0ad9e9 nodeName:}" failed. No retries permitted until 2026-05-28 00:32:24.457434595 +0000 UTC m=+1253.785100683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-84bxj" (UID: "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 00:32:23.957858 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:32:23.957485    2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 00:32:23.957858 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:32:23.957548    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-ui-tls podName:d066b84c-9cf1-4386-821a-9b0c9d0ad9e9 nodeName:}" failed. No retries permitted until 2026-05-28 00:32:24.457508898 +0000 UTC m=+1253.785174988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-84bxj" (UID: "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 00:32:23.957961 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.957870    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.960809 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.960777    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.960920 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.960836    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:23.968290 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:23.968258    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k24j\" (UniqueName: \"kubernetes.io/projected/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-kube-api-access-9k24j\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:24.461358 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.461280    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:24.461358 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.461346    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:24.463860 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.463832    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:24.463860 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.463852    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-84bxj\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:24.728345 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.728258    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:24.741679 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.741650    2560 generic.go:358] "Generic (PLEG): container finished" podID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerID="688957703c62f2c61593ba5cce8054db17fbe9a9891aced251a020a58e73ecfb" exitCode=0
May 28 00:32:24.741679 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.741674    2560 generic.go:358] "Generic (PLEG): container finished" podID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerID="00825903c87c970911673e0aaf5ca9729fa305ca9135eeece75a5a27c2dfc02f" exitCode=0
May 28 00:32:24.741852 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.741720    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerDied","Data":"688957703c62f2c61593ba5cce8054db17fbe9a9891aced251a020a58e73ecfb"}
May 28 00:32:24.741852 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.741752    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerDied","Data":"00825903c87c970911673e0aaf5ca9729fa305ca9135eeece75a5a27c2dfc02f"}
May 28 00:32:24.858320 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:24.858286    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"]
May 28 00:32:24.861257 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:32:24.861228    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd066b84c_9cf1_4386_821a_9b0c9d0ad9e9.slice/crio-f63a74fe0592f5c639d704a23ae8f66088b0fdf8e2113e697995fbc2cf575dbd WatchSource:0}: Error finding container f63a74fe0592f5c639d704a23ae8f66088b0fdf8e2113e697995fbc2cf575dbd: Status 404 returned error can't find the container with id f63a74fe0592f5c639d704a23ae8f66088b0fdf8e2113e697995fbc2cf575dbd
May 28 00:32:25.745595 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:25.745554    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerStarted","Data":"e9b432339c5004a4956fcbc197eb6a498155e46debc4943d06e935fc2f16c163"}
May 28 00:32:25.745595 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:25.745598    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerStarted","Data":"f63a74fe0592f5c639d704a23ae8f66088b0fdf8e2113e697995fbc2cf575dbd"}
May 28 00:32:29.761175 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:29.761087    2560 generic.go:358] "Generic (PLEG): container finished" podID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerID="e9b432339c5004a4956fcbc197eb6a498155e46debc4943d06e935fc2f16c163" exitCode=0
May 28 00:32:29.761175 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:29.761162    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerDied","Data":"e9b432339c5004a4956fcbc197eb6a498155e46debc4943d06e935fc2f16c163"}
May 28 00:32:30.300486 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:30.300432    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused"
May 28 00:32:30.567791 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:30.567708    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused"
May 28 00:32:30.766163 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:30.766127    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerStarted","Data":"e6f74aeb5c1cbbf58a0710f96f174cd3e0ab17d8fe6b3dfd0da2ddcbaf632123"}
May 28 00:32:35.780061 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:35.780025    2560 generic.go:358] "Generic (PLEG): container finished" podID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerID="e6f74aeb5c1cbbf58a0710f96f174cd3e0ab17d8fe6b3dfd0da2ddcbaf632123" exitCode=0
May 28 00:32:35.780507 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:35.780073    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerDied","Data":"e6f74aeb5c1cbbf58a0710f96f174cd3e0ab17d8fe6b3dfd0da2ddcbaf632123"}
May 28 00:32:36.787094 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:36.787048    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerStarted","Data":"410ec09cfaf21666197762cb89be0669373fd4ae61c1355a0e7b74a9da5b7806"}
May 28 00:32:36.787586 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:36.787105    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerStarted","Data":"8ce7f5c4864b4277576fbe48b9deebe37db86751d6e0e876a28e5044f0868e1b"}
May 28 00:32:36.787586 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:36.787122    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerStarted","Data":"79effdb11252770b9fed9c98fcf766cf81c4377c5d9ae2080feddf9afc615e44"}
May 28 00:32:36.787586 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:36.787136    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerStarted","Data":"4da5cb221f7110affbca9c91985a9ff0cc599eedfcc94efebb933f22e407d921"}
May 28 00:32:36.811080 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:36.811020    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podStartSLOduration=13.81100182 podStartE2EDuration="13.81100182s" podCreationTimestamp="2026-05-28 00:32:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:32:36.809039414 +0000 UTC m=+1266.136705523" watchObservedRunningTime="2026-05-28 00:32:36.81100182 +0000 UTC m=+1266.138667941"
May 28 00:32:39.729379 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:39.729337    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:39.729856 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:39.729392    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:39.729856 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:39.729407    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:39.730239 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:39.730199    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:39.731436 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:39.731391    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 28 00:32:39.731575 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:39.731428    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused"
May 28 00:32:39.731575 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:39.731504    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused"
May 28 00:32:39.731705 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:39.731623    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 28 00:32:40.301168 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:40.301123    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused"
May 28 00:32:40.568664 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:40.568541    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused"
May 28 00:32:42.729838 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.729803    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.730273 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.729905    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.730273 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.730185    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.730759 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.730730    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.730898 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.730767    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.730898 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.730781    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.730898 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.730872    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.731012 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.730967    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.731167 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.731153    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.734288 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.734269    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.806803 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.806782    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:42.809645 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:42.809622    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:32:50.300165 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:50.300114    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused"
May 28 00:32:50.300570 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:50.300246    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:32:50.567671 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:50.567576    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused"
May 28 00:32:50.567805 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:50.567704    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:32:53.836895 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:53.836865    2560 generic.go:358] "Generic (PLEG): container finished" podID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerID="949c1ac98ab049376bd27e7cca37fe8d02ecc1e00925d85e401b14ccb218794e" exitCode=137
May 28 00:32:53.836895 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:53.836891    2560 generic.go:358] "Generic (PLEG): container finished" podID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerID="551a16e85642df153b892ed3909952daa28718dba78606ac3a9d6ef001f4b531" exitCode=137
May 28 00:32:53.837287 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:53.836940    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerDied","Data":"949c1ac98ab049376bd27e7cca37fe8d02ecc1e00925d85e401b14ccb218794e"}
May 28 00:32:53.837287 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:53.836973    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerDied","Data":"551a16e85642df153b892ed3909952daa28718dba78606ac3a9d6ef001f4b531"}
May 28 00:32:54.144282 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.144259    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:32:54.302658 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.302623    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf8bt\" (UniqueName: \"kubernetes.io/projected/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-kube-api-access-qf8bt\") pod \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") "
May 28 00:32:54.302815 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.302671    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-online-tls\") pod \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") "
May 28 00:32:54.302815 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.302710    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-ui-tls\") pod \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") "
May 28 00:32:54.302815 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.302750    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-feast-data\") pod \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") "
May 28 00:32:54.302815 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.302789    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-offline-tls\") pod \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") "
May 28 00:32:54.302815 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.302812    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-registry-tls\") pod \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\" (UID: \"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d\") "
May 28 00:32:54.303332 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.303302    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-feast-data" (OuterVolumeSpecName: "feast-data") pod "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" (UID: "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:32:54.305188 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.305164    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" (UID: "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:32:54.305405 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.305381    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-kube-api-access-qf8bt" (OuterVolumeSpecName: "kube-api-access-qf8bt") pod "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" (UID: "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d"). InnerVolumeSpecName "kube-api-access-qf8bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:32:54.305505 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.305400    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" (UID: "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:32:54.305505 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.305459    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-online-tls" (OuterVolumeSpecName: "online-tls") pod "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" (UID: "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:32:54.305505 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.305473    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" (UID: "62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:32:54.404152 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.404095    2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-offline-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:32:54.404152 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.404119    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-registry-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:32:54.404279 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.404227    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf8bt\" (UniqueName: \"kubernetes.io/projected/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-kube-api-access-qf8bt\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:32:54.404313 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.404299    2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-online-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:32:54.404344 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.404316    2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-ui-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:32:54.404344 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.404329    2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d-feast-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:32:54.841907 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.841873    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f" event={"ID":"62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d","Type":"ContainerDied","Data":"dbfab8c8690e661a63091d55d0523333641e07c2bdfe575d9da5ec001b932f10"}
May 28 00:32:54.842313 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.841921    2560 scope.go:117] "RemoveContainer" containerID="688957703c62f2c61593ba5cce8054db17fbe9a9891aced251a020a58e73ecfb"
May 28 00:32:54.842386 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.842350    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"
May 28 00:32:54.855873 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.855850    2560 scope.go:117] "RemoveContainer" containerID="949c1ac98ab049376bd27e7cca37fe8d02ecc1e00925d85e401b14ccb218794e"
May 28 00:32:54.862609 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.862590    2560 scope.go:117] "RemoveContainer" containerID="00825903c87c970911673e0aaf5ca9729fa305ca9135eeece75a5a27c2dfc02f"
May 28 00:32:54.868622 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.868593    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"]
May 28 00:32:54.869919 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.869901    2560 scope.go:117] "RemoveContainer" containerID="551a16e85642df153b892ed3909952daa28718dba78606ac3a9d6ef001f4b531"
May 28 00:32:54.872678 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.872654    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-wds2f"]
May 28 00:32:54.876748 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.876732    2560 scope.go:117] "RemoveContainer" containerID="6cd84ac2986c76cbeab9205e62be2dfa7f76e19a7c645f309381e31d7e59b531"
May 28 00:32:54.883197 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:54.883183    2560 scope.go:117] "RemoveContainer" containerID="d918ef00ccaa1562ffb62a355cd60683e38836acd82039f885710d1f2bcd01c2"
May 28 00:32:55.260561 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:32:55.260475    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" path="/var/lib/kubelet/pods/62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d/volumes"
May 28 00:33:45.639709 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.639629    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"]
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.639937    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="feast-init"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.639950    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="feast-init"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.639962    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="registry"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.639971    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="registry"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.639984    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="online"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.639994    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="online"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640002    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="ui"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640007    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="ui"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640019    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="offline"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640027    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="offline"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640038    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="feast-apply"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640046    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="feast-apply"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640100    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="offline"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640118    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="online"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640125    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="ui"
May 28 00:33:45.640123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.640131    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="62bf816e-ee87-4c31-a2ed-5e73ad2f9d9d" containerName="registry"
May 28 00:33:45.644731 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.644711    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.647418 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.647392    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 00:33:45.647418 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.647416    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 00:33:45.647623 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.647414    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-kdrr8\""
May 28 00:33:45.647623 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.647472    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 00:33:45.647623 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.647477    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 00:33:45.647623 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.647559    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 00:33:45.647784 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.647664    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 00:33:45.657021 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.657001    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"]
May 28 00:33:45.668268 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.668245    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.668375 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.668271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.668375 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.668295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk52j\" (UniqueName: \"kubernetes.io/projected/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-kube-api-access-fk52j\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.668485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.668392    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.668485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.668426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.668590 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.668493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.768952 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.768920    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.769068 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.768957    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.769068 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.768974    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.769068 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.769000    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fk52j\" (UniqueName: \"kubernetes.io/projected/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-kube-api-access-fk52j\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.769068 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.769041    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.769213 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.769069    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.769213 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:33:45.769099    2560 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:33:45.769213 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:33:45.769163    2560 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 00:33:45.769213 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:33:45.769180    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-online-tls podName:19fe9ed5-60c6-4197-a8d1-9c2f281dc354 nodeName:}" failed. No retries permitted until 2026-05-28 00:33:46.269158659 +0000 UTC m=+1335.596824757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-online-tls") pod "feast-simple-feast-remote-setup-5cff94746b-l48qn" (UID: "19fe9ed5-60c6-4197-a8d1-9c2f281dc354") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:33:45.769213 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:33:45.769204    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-offline-tls podName:19fe9ed5-60c6-4197-a8d1-9c2f281dc354 nodeName:}" failed. No retries permitted until 2026-05-28 00:33:46.269194026 +0000 UTC m=+1335.596860116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-offline-tls") pod "feast-simple-feast-remote-setup-5cff94746b-l48qn" (UID: "19fe9ed5-60c6-4197-a8d1-9c2f281dc354") : secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 00:33:45.769394 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.769320    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.769717 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.769699    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.771423 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.771401    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:45.777687 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:45.777662    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk52j\" (UniqueName: \"kubernetes.io/projected/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-kube-api-access-fk52j\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:46.273721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:46.273692    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:46.273881 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:46.273754    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:46.276318 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:46.276284    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:46.276318 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:46.276301    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-l48qn\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:46.554229 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:46.554190    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:33:46.681943 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:46.681916    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"]
May 28 00:33:46.684283 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:33:46.684253    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19fe9ed5_60c6_4197_a8d1_9c2f281dc354.slice/crio-07102fef00bc76512207e23ead5320e237bf3c5437781dfd27755ce6152b8865 WatchSource:0}: Error finding container 07102fef00bc76512207e23ead5320e237bf3c5437781dfd27755ce6152b8865: Status 404 returned error can't find the container with id 07102fef00bc76512207e23ead5320e237bf3c5437781dfd27755ce6152b8865
May 28 00:33:46.989500 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:46.989407    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerStarted","Data":"2e0a03652dd50556087605f11cbbc528a360e52c0a3ccd61eb9ee4a3d6f0dc4e"}
May 28 00:33:46.989500 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:46.989456    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerStarted","Data":"07102fef00bc76512207e23ead5320e237bf3c5437781dfd27755ce6152b8865"}
May 28 00:33:52.004025 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:52.003990    2560 generic.go:358] "Generic (PLEG): container finished" podID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerID="2e0a03652dd50556087605f11cbbc528a360e52c0a3ccd61eb9ee4a3d6f0dc4e" exitCode=0
May 28 00:33:52.004466 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:52.004065    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerDied","Data":"2e0a03652dd50556087605f11cbbc528a360e52c0a3ccd61eb9ee4a3d6f0dc4e"}
May 28 00:33:53.008165 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:53.008124    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerStarted","Data":"104a3597574b961b25ae656f779c19d986eefc83153b757253764428744aa62c"}
May 28 00:33:58.025004 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:58.024971    2560 generic.go:358] "Generic (PLEG): container finished" podID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerID="104a3597574b961b25ae656f779c19d986eefc83153b757253764428744aa62c" exitCode=0
May 28 00:33:58.025366 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:58.025038    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerDied","Data":"104a3597574b961b25ae656f779c19d986eefc83153b757253764428744aa62c"}
May 28 00:33:59.031144 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:59.031100    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerStarted","Data":"b3689afbbc2dcb69a9fdc56fd9bffd8c2bd191f4dd8af1c6e4f520936721b77e"}
May 28 00:33:59.031144 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:59.031146    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerStarted","Data":"e2f55f0dfef3f8c836a643c0d1e8273b6d9b138beb4c6d0bd69af560455edb5c"}
May 28 00:33:59.031585 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:59.031160    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerStarted","Data":"0edcc27594dfe05ddffad7d5044bf0a081c0245fdee22ad3d0d2fb0e4ea0dfe7"}
May 28 00:33:59.055529 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:33:59.055451    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podStartSLOduration=14.055433057 podStartE2EDuration="14.055433057s" podCreationTimestamp="2026-05-28 00:33:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:33:59.054401374 +0000 UTC m=+1348.382067482" watchObservedRunningTime="2026-05-28 00:33:59.055433057 +0000 UTC m=+1348.383099166"
May 28 00:34:01.555069 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:01.555027    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:01.555582 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:01.555078    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:01.555582 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:01.555091    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:01.557028 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:01.556976    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 28 00:34:01.557028 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:01.557000    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 28 00:34:01.557028 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:01.556971    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused"
May 28 00:34:04.555304 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:04.555267    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:04.555731 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:04.555571    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:04.555905 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:04.555884    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:04.555982 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:04.555914    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:04.556063 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:04.556050    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:04.556123 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:04.556111    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:04.559628 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:04.559602    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:05.049840 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:05.049815    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:34:05.052738 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:34:05.052718    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:35:06.842492 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:06.842455    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"]
May 28 00:35:06.843089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:06.842916    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="online" containerID="cri-o://0edcc27594dfe05ddffad7d5044bf0a081c0245fdee22ad3d0d2fb0e4ea0dfe7" gracePeriod=30
May 28 00:35:06.843236 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:06.843207    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="offline" containerID="cri-o://e2f55f0dfef3f8c836a643c0d1e8273b6d9b138beb4c6d0bd69af560455edb5c" gracePeriod=30
May 28 00:35:06.843465 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:06.843204    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="ui" containerID="cri-o://b3689afbbc2dcb69a9fdc56fd9bffd8c2bd191f4dd8af1c6e4f520936721b77e" gracePeriod=30
May 28 00:35:07.013327 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:07.013286    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"]
May 28 00:35:07.014168 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:07.013792    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="registry" containerID="cri-o://4da5cb221f7110affbca9c91985a9ff0cc599eedfcc94efebb933f22e407d921" gracePeriod=30
May 28 00:35:07.014168 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:07.013941    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="online" containerID="cri-o://79effdb11252770b9fed9c98fcf766cf81c4377c5d9ae2080feddf9afc615e44" gracePeriod=30
May 28 00:35:07.014168 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:07.013959    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="ui" containerID="cri-o://410ec09cfaf21666197762cb89be0669373fd4ae61c1355a0e7b74a9da5b7806" gracePeriod=30
May 28 00:35:07.014168 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:07.013943    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="offline" containerID="cri-o://8ce7f5c4864b4277576fbe48b9deebe37db86751d6e0e876a28e5044f0868e1b" gracePeriod=30
May 28 00:35:08.231226 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:08.231191    2560 generic.go:358] "Generic (PLEG): container finished" podID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerID="410ec09cfaf21666197762cb89be0669373fd4ae61c1355a0e7b74a9da5b7806" exitCode=0
May 28 00:35:08.231226 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:08.231218    2560 generic.go:358] "Generic (PLEG): container finished" podID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerID="79effdb11252770b9fed9c98fcf766cf81c4377c5d9ae2080feddf9afc615e44" exitCode=0
May 28 00:35:08.231713 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:08.231258    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerDied","Data":"410ec09cfaf21666197762cb89be0669373fd4ae61c1355a0e7b74a9da5b7806"}
May 28 00:35:08.231713 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:08.231293    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerDied","Data":"79effdb11252770b9fed9c98fcf766cf81c4377c5d9ae2080feddf9afc615e44"}
May 28 00:35:08.233449 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:08.233427    2560 generic.go:358] "Generic (PLEG): container finished" podID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerID="b3689afbbc2dcb69a9fdc56fd9bffd8c2bd191f4dd8af1c6e4f520936721b77e" exitCode=0
May 28 00:35:08.233449 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:08.233446    2560 generic.go:358] "Generic (PLEG): container finished" podID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerID="0edcc27594dfe05ddffad7d5044bf0a081c0245fdee22ad3d0d2fb0e4ea0dfe7" exitCode=0
May 28 00:35:08.233618 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:08.233485    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerDied","Data":"b3689afbbc2dcb69a9fdc56fd9bffd8c2bd191f4dd8af1c6e4f520936721b77e"}
May 28 00:35:08.233618 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:08.233503    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerDied","Data":"0edcc27594dfe05ddffad7d5044bf0a081c0245fdee22ad3d0d2fb0e4ea0dfe7"}
May 28 00:35:12.731752 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:12.731656    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 28 00:35:12.807419 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:12.807381    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 28 00:35:14.556282 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:14.556234    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 28 00:35:15.050585 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:15.050541    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 28 00:35:22.730979 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:22.730935    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 28 00:35:22.807896 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:22.807854    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 28 00:35:24.556741 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:24.556690    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 28 00:35:25.050538 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:25.050477    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 28 00:35:32.731275 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:32.731235    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused"
May 28 00:35:32.731740 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:32.731369    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:35:32.807646 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:32.807608    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused"
May 28 00:35:32.807819 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:32.807725    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:35:34.556097 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:34.556056    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused"
May 28 00:35:34.556500 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:34.556199    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:35:35.050452 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:35.050411    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused"
May 28 00:35:35.050607 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:35.050581    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:35:37.309682 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.309652    2560 generic.go:358] "Generic (PLEG): container finished" podID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerID="8ce7f5c4864b4277576fbe48b9deebe37db86751d6e0e876a28e5044f0868e1b" exitCode=137
May 28 00:35:37.309682 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.309677    2560 generic.go:358] "Generic (PLEG): container finished" podID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerID="4da5cb221f7110affbca9c91985a9ff0cc599eedfcc94efebb933f22e407d921" exitCode=137
May 28 00:35:37.310094 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.309731    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerDied","Data":"8ce7f5c4864b4277576fbe48b9deebe37db86751d6e0e876a28e5044f0868e1b"}
May 28 00:35:37.310094 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.309771    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerDied","Data":"4da5cb221f7110affbca9c91985a9ff0cc599eedfcc94efebb933f22e407d921"}
May 28 00:35:37.311758 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.311732    2560 generic.go:358] "Generic (PLEG): container finished" podID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerID="e2f55f0dfef3f8c836a643c0d1e8273b6d9b138beb4c6d0bd69af560455edb5c" exitCode=137
May 28 00:35:37.311817 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.311782    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerDied","Data":"e2f55f0dfef3f8c836a643c0d1e8273b6d9b138beb4c6d0bd69af560455edb5c"}
May 28 00:35:37.503874 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.503851    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:35:37.580419 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.580336    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk52j\" (UniqueName: \"kubernetes.io/projected/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-kube-api-access-fk52j\") pod \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") "
May 28 00:35:37.580419 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.580383    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-online-tls\") pod \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") "
May 28 00:35:37.580419 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.580399    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-ui-tls\") pod \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") "
May 28 00:35:37.580709 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.580428    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-offline-tls\") pod \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") "
May 28 00:35:37.580709 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.580450    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-feast-data\") pod \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") "
May 28 00:35:37.580709 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.580561    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-registry-tls\") pod \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\" (UID: \"19fe9ed5-60c6-4197-a8d1-9c2f281dc354\") "
May 28 00:35:37.580958 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.580926    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-feast-data" (OuterVolumeSpecName: "feast-data") pod "19fe9ed5-60c6-4197-a8d1-9c2f281dc354" (UID: "19fe9ed5-60c6-4197-a8d1-9c2f281dc354"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:35:37.581132 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.581093    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "19fe9ed5-60c6-4197-a8d1-9c2f281dc354" (UID: "19fe9ed5-60c6-4197-a8d1-9c2f281dc354"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:35:37.582801 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.582777    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-kube-api-access-fk52j" (OuterVolumeSpecName: "kube-api-access-fk52j") pod "19fe9ed5-60c6-4197-a8d1-9c2f281dc354" (UID: "19fe9ed5-60c6-4197-a8d1-9c2f281dc354"). InnerVolumeSpecName "kube-api-access-fk52j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:35:37.582919 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.582865    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-online-tls" (OuterVolumeSpecName: "online-tls") pod "19fe9ed5-60c6-4197-a8d1-9c2f281dc354" (UID: "19fe9ed5-60c6-4197-a8d1-9c2f281dc354"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:35:37.583117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.583099    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "19fe9ed5-60c6-4197-a8d1-9c2f281dc354" (UID: "19fe9ed5-60c6-4197-a8d1-9c2f281dc354"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:35:37.583280 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.583264    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "19fe9ed5-60c6-4197-a8d1-9c2f281dc354" (UID: "19fe9ed5-60c6-4197-a8d1-9c2f281dc354"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:35:37.653139 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.653109    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:35:37.681264 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.681235    2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-online-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.681264 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.681260    2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-ui-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.681264 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.681270    2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-offline-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.681461 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.681278    2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-feast-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.681461 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.681286    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-registry-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.681461 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.681295    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk52j\" (UniqueName: \"kubernetes.io/projected/19fe9ed5-60c6-4197-a8d1-9c2f281dc354-kube-api-access-fk52j\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.781698 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.781669    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-offline-tls\") pod \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") "
May 28 00:35:37.781698 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.781699    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k24j\" (UniqueName: \"kubernetes.io/projected/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-kube-api-access-9k24j\") pod \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") "
May 28 00:35:37.781907 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.781728    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-ui-tls\") pod \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") "
May 28 00:35:37.781907 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.781790    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-feast-data\") pod \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") "
May 28 00:35:37.781907 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.781818    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-online-tls\") pod \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") "
May 28 00:35:37.781907 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.781842    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-registry-tls\") pod \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\" (UID: \"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9\") "
May 28 00:35:37.782348 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.782318    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-feast-data" (OuterVolumeSpecName: "feast-data") pod "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" (UID: "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:35:37.784084 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.784054    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" (UID: "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:35:37.784175 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.784085    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" (UID: "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:35:37.784175 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.784101    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-kube-api-access-9k24j" (OuterVolumeSpecName: "kube-api-access-9k24j") pod "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" (UID: "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9"). InnerVolumeSpecName "kube-api-access-9k24j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:35:37.784531 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.784491    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" (UID: "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:35:37.784531 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.784502    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-online-tls" (OuterVolumeSpecName: "online-tls") pod "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" (UID: "d066b84c-9cf1-4386-821a-9b0c9d0ad9e9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:35:37.882995 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.882973    2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-offline-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.882995 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.882994    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k24j\" (UniqueName: \"kubernetes.io/projected/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-kube-api-access-9k24j\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.883105 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.883005    2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-ui-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.883105 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.883013    2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-feast-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.883105 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.883022    2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-online-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:37.883105 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:37.883030    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9-registry-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:38.316888 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.316857    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"
May 28 00:35:38.317384 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.316861    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn" event={"ID":"19fe9ed5-60c6-4197-a8d1-9c2f281dc354","Type":"ContainerDied","Data":"07102fef00bc76512207e23ead5320e237bf3c5437781dfd27755ce6152b8865"}
May 28 00:35:38.317384 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.316987    2560 scope.go:117] "RemoveContainer" containerID="b3689afbbc2dcb69a9fdc56fd9bffd8c2bd191f4dd8af1c6e4f520936721b77e"
May 28 00:35:38.319468 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.319443    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj" event={"ID":"d066b84c-9cf1-4386-821a-9b0c9d0ad9e9","Type":"ContainerDied","Data":"f63a74fe0592f5c639d704a23ae8f66088b0fdf8e2113e697995fbc2cf575dbd"}
May 28 00:35:38.319595 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.319480    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"
May 28 00:35:38.326399 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.326379    2560 scope.go:117] "RemoveContainer" containerID="e2f55f0dfef3f8c836a643c0d1e8273b6d9b138beb4c6d0bd69af560455edb5c"
May 28 00:35:38.333478 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.333463    2560 scope.go:117] "RemoveContainer" containerID="0edcc27594dfe05ddffad7d5044bf0a081c0245fdee22ad3d0d2fb0e4ea0dfe7"
May 28 00:35:38.339411 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.339388    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"]
May 28 00:35:38.340832 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.340813    2560 scope.go:117] "RemoveContainer" containerID="104a3597574b961b25ae656f779c19d986eefc83153b757253764428744aa62c"
May 28 00:35:38.343354 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.343334    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-l48qn"]
May 28 00:35:38.347804 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.347784    2560 scope.go:117] "RemoveContainer" containerID="2e0a03652dd50556087605f11cbbc528a360e52c0a3ccd61eb9ee4a3d6f0dc4e"
May 28 00:35:38.354304 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.354284    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"]
May 28 00:35:38.354498 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.354483    2560 scope.go:117] "RemoveContainer" containerID="410ec09cfaf21666197762cb89be0669373fd4ae61c1355a0e7b74a9da5b7806"
May 28 00:35:38.358823 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.358806    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-84bxj"]
May 28 00:35:38.361787 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.361770    2560 scope.go:117] "RemoveContainer" containerID="8ce7f5c4864b4277576fbe48b9deebe37db86751d6e0e876a28e5044f0868e1b"
May 28 00:35:38.367949 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.367933    2560 scope.go:117] "RemoveContainer" containerID="79effdb11252770b9fed9c98fcf766cf81c4377c5d9ae2080feddf9afc615e44"
May 28 00:35:38.374138 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.374123    2560 scope.go:117] "RemoveContainer" containerID="4da5cb221f7110affbca9c91985a9ff0cc599eedfcc94efebb933f22e407d921"
May 28 00:35:38.380268 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.380254    2560 scope.go:117] "RemoveContainer" containerID="e6f74aeb5c1cbbf58a0710f96f174cd3e0ab17d8fe6b3dfd0da2ddcbaf632123"
May 28 00:35:38.386580 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:38.386561    2560 scope.go:117] "RemoveContainer" containerID="e9b432339c5004a4956fcbc197eb6a498155e46debc4943d06e935fc2f16c163"
May 28 00:35:39.260257 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:39.260224    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" path="/var/lib/kubelet/pods/19fe9ed5-60c6-4197-a8d1-9c2f281dc354/volumes"
May 28 00:35:39.260774 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:39.260760    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" path="/var/lib/kubelet/pods/d066b84c-9cf1-4386-821a-9b0c9d0ad9e9/volumes"
May 28 00:35:56.089119 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.089075    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"]
May 28 00:35:56.089711 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.089386    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx" podUID="dbde35c5-a5df-43a2-a7ba-1cd28e0443b8" containerName="manager" containerID="cri-o://4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff" gracePeriod=10
May 28 00:35:56.331688 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.331664    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"
May 28 00:35:56.367575 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.367461    2560 generic.go:358] "Generic (PLEG): container finished" podID="dbde35c5-a5df-43a2-a7ba-1cd28e0443b8" containerID="4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff" exitCode=0
May 28 00:35:56.367575 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.367549    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"
May 28 00:35:56.367786 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.367570    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx" event={"ID":"dbde35c5-a5df-43a2-a7ba-1cd28e0443b8","Type":"ContainerDied","Data":"4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff"}
May 28 00:35:56.367786 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.367618    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx" event={"ID":"dbde35c5-a5df-43a2-a7ba-1cd28e0443b8","Type":"ContainerDied","Data":"15fde7c67f19c690e5dc030afe38874123b752554a437e8308eb42b7b31331c6"}
May 28 00:35:56.367786 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.367640    2560 scope.go:117] "RemoveContainer" containerID="4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff"
May 28 00:35:56.375081 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.375062    2560 scope.go:117] "RemoveContainer" containerID="4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff"
May 28 00:35:56.375344 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:35:56.375323    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff\": container with ID starting with 4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff not found: ID does not exist" containerID="4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff"
May 28 00:35:56.375428 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.375350    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff"} err="failed to get container status \"4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff\": rpc error: code = NotFound desc = could not find container \"4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff\": container with ID starting with 4a90db639503b598052744253b164a2e61fbedeaf5392278239a12376ba63aff not found: ID does not exist"
May 28 00:35:56.422215 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.422188    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqwhz\" (UniqueName: \"kubernetes.io/projected/dbde35c5-a5df-43a2-a7ba-1cd28e0443b8-kube-api-access-xqwhz\") pod \"dbde35c5-a5df-43a2-a7ba-1cd28e0443b8\" (UID: \"dbde35c5-a5df-43a2-a7ba-1cd28e0443b8\") "
May 28 00:35:56.424335 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.424316    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbde35c5-a5df-43a2-a7ba-1cd28e0443b8-kube-api-access-xqwhz" (OuterVolumeSpecName: "kube-api-access-xqwhz") pod "dbde35c5-a5df-43a2-a7ba-1cd28e0443b8" (UID: "dbde35c5-a5df-43a2-a7ba-1cd28e0443b8"). InnerVolumeSpecName "kube-api-access-xqwhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:35:56.523354 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.523327    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwhz\" (UniqueName: \"kubernetes.io/projected/dbde35c5-a5df-43a2-a7ba-1cd28e0443b8-kube-api-access-xqwhz\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:35:56.688204 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.688174    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"]
May 28 00:35:56.691863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:56.691839    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-h54dx"]
May 28 00:35:57.260102 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:35:57.260068    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbde35c5-a5df-43a2-a7ba-1cd28e0443b8" path="/var/lib/kubelet/pods/dbde35c5-a5df-43a2-a7ba-1cd28e0443b8/volumes"
May 28 00:36:00.729396 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729367    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"]
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729741    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="registry"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729760    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="registry"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729774    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="ui"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729782    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="ui"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729795    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="offline"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729803    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="offline"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729813    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="feast-init"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729822    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="feast-init"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729831    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="feast-apply"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729838    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="feast-apply"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729846    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="feast-apply"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729854    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="feast-apply"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729864    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="feast-init"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729871    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="feast-init"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729883    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="online"
May 28 00:36:00.729878 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729890    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="online"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729901    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="offline"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729909    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="offline"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729917    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="ui"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729924    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="ui"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729937    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="online"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729944    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="online"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729953    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbde35c5-a5df-43a2-a7ba-1cd28e0443b8" containerName="manager"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.729961    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbde35c5-a5df-43a2-a7ba-1cd28e0443b8" containerName="manager"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.730011    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="registry"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.730022    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="online"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.730031    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="ui"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.730040    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbde35c5-a5df-43a2-a7ba-1cd28e0443b8" containerName="manager"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.730049    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d066b84c-9cf1-4386-821a-9b0c9d0ad9e9" containerName="offline"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.730059    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="online"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.730068    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="ui"
May 28 00:36:00.730400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.730077    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="19fe9ed5-60c6-4197-a8d1-9c2f281dc354" containerName="offline"
May 28 00:36:00.733956 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.733941    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"
May 28 00:36:00.737721 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.737698    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-2xpq9\""
May 28 00:36:00.739445 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.739425    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 28 00:36:00.739580 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.739505    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 28 00:36:00.761885 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.761864    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"]
May 28 00:36:00.851230 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.851193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgcsh\" (UniqueName: \"kubernetes.io/projected/a40c983d-1e17-4605-b96d-5fef4e21e612-kube-api-access-lgcsh\") pod \"feast-operator-controller-manager-cf99db75-p7hzm\" (UID: \"a40c983d-1e17-4605-b96d-5fef4e21e612\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"
May 28 00:36:00.952190 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.952163    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lgcsh\" (UniqueName: \"kubernetes.io/projected/a40c983d-1e17-4605-b96d-5fef4e21e612-kube-api-access-lgcsh\") pod \"feast-operator-controller-manager-cf99db75-p7hzm\" (UID: \"a40c983d-1e17-4605-b96d-5fef4e21e612\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"
May 28 00:36:00.960599 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:00.960580    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgcsh\" (UniqueName: \"kubernetes.io/projected/a40c983d-1e17-4605-b96d-5fef4e21e612-kube-api-access-lgcsh\") pod \"feast-operator-controller-manager-cf99db75-p7hzm\" (UID: \"a40c983d-1e17-4605-b96d-5fef4e21e612\") " pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"
May 28 00:36:01.043911 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:01.043888    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"
May 28 00:36:01.169165 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:01.169134    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"]
May 28 00:36:01.171998 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:36:01.171962    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40c983d_1e17_4605_b96d_5fef4e21e612.slice/crio-ec18c4e2cac4f1e387ab33e3116aa5658b2c5c423d3fd5dd94982fc541a0ccbc WatchSource:0}: Error finding container ec18c4e2cac4f1e387ab33e3116aa5658b2c5c423d3fd5dd94982fc541a0ccbc: Status 404 returned error can't find the container with id ec18c4e2cac4f1e387ab33e3116aa5658b2c5c423d3fd5dd94982fc541a0ccbc
May 28 00:36:01.173674 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:01.173658    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:36:01.384192 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:01.384099    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm" event={"ID":"a40c983d-1e17-4605-b96d-5fef4e21e612","Type":"ContainerStarted","Data":"511264e3293f7df796e50efba2b08f334049a68ecb7d8bab97231973a6b2ac66"}
May 28 00:36:01.384192 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:01.384142    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm" event={"ID":"a40c983d-1e17-4605-b96d-5fef4e21e612","Type":"ContainerStarted","Data":"ec18c4e2cac4f1e387ab33e3116aa5658b2c5c423d3fd5dd94982fc541a0ccbc"}
May 28 00:36:01.384405 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:01.384210    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"
May 28 00:36:01.403675 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:01.403633    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm" podStartSLOduration=1.40362012 podStartE2EDuration="1.40362012s" podCreationTimestamp="2026-05-28 00:36:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:36:01.402611918 +0000 UTC m=+1470.730278026" watchObservedRunningTime="2026-05-28 00:36:01.40362012 +0000 UTC m=+1470.731286228"
May 28 00:36:12.389327 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:12.389300    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-cf99db75-p7hzm"
May 28 00:36:18.808665 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.808621    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"]
May 28 00:36:18.812197 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.812179    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:18.814304 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.814279    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 00:36:18.814420 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.814320    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 00:36:18.814420 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.814337    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-sfsgm\""
May 28 00:36:18.814420 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.814337    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 00:36:18.814593 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.814551    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 00:36:18.815039 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.815021    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 00:36:18.815125 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.815076    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 00:36:18.830275 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.830251    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"]
May 28 00:36:18.984633 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.984596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea76d9a2-64e4-4adc-993b-cf7ef643b557-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:18.984775 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.984639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:18.984775 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.984665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/ea76d9a2-64e4-4adc-993b-cf7ef643b557-kube-api-access-7k2bl\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:18.984775 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.984722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:18.984775 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.984766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:18.984910 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:18.984810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.085346 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.085268    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.085346 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.085318    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.085346 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.085343    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea76d9a2-64e4-4adc-993b-cf7ef643b557-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.085603 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.085358    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.085603 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.085376    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/ea76d9a2-64e4-4adc-993b-cf7ef643b557-kube-api-access-7k2bl\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.085603 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.085400    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.085851 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.085823    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea76d9a2-64e4-4adc-993b-cf7ef643b557-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.087973 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.087953    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.088097 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.088074    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.088162 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.088144    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.088197 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.088143    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.093348 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.093330    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/ea76d9a2-64e4-4adc-993b-cf7ef643b557-kube-api-access-7k2bl\") pod \"feast-simple-feast-setup-664fcfcc58-lghgs\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.121392 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.121366    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:19.250278 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.250248    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"]
May 28 00:36:19.253300 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:36:19.253270    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea76d9a2_64e4_4adc_993b_cf7ef643b557.slice/crio-41a02396491d8462c9343838e0ad4798d523654893e482e4a3abf55be926fa3a WatchSource:0}: Error finding container 41a02396491d8462c9343838e0ad4798d523654893e482e4a3abf55be926fa3a: Status 404 returned error can't find the container with id 41a02396491d8462c9343838e0ad4798d523654893e482e4a3abf55be926fa3a
May 28 00:36:19.433378 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.433285    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerStarted","Data":"4ea30029ee891f2cbf3ce2e4436e814d22cbfc385f15c1600c4a964ceea6f61c"}
May 28 00:36:19.433378 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:19.433322    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerStarted","Data":"41a02396491d8462c9343838e0ad4798d523654893e482e4a3abf55be926fa3a"}
May 28 00:36:24.448017 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:24.447985    2560 generic.go:358] "Generic (PLEG): container finished" podID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerID="4ea30029ee891f2cbf3ce2e4436e814d22cbfc385f15c1600c4a964ceea6f61c" exitCode=0
May 28 00:36:24.448427 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:24.448058    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerDied","Data":"4ea30029ee891f2cbf3ce2e4436e814d22cbfc385f15c1600c4a964ceea6f61c"}
May 28 00:36:25.453244 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:25.453210    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerStarted","Data":"75ba5f63a7e8c526ad69f762c8b0734507ed3e08baba7041d993df44a46748a6"}
May 28 00:36:30.472938 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:30.472902    2560 generic.go:358] "Generic (PLEG): container finished" podID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerID="75ba5f63a7e8c526ad69f762c8b0734507ed3e08baba7041d993df44a46748a6" exitCode=0
May 28 00:36:30.473327 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:30.472977    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerDied","Data":"75ba5f63a7e8c526ad69f762c8b0734507ed3e08baba7041d993df44a46748a6"}
May 28 00:36:31.480533 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:31.480472    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerStarted","Data":"f6991528a5ff8c82756b9b33755f576631463ead81a6fdaa299a4453a6a24be4"}
May 28 00:36:31.480991 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:31.480542    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerStarted","Data":"4a9bd4895a97c3c05ac33b6209036893940c23d65ca3a5cb46594a95b4cdda39"}
May 28 00:36:31.480991 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:31.480559    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerStarted","Data":"778558ea3c1ac9cf603e74e536aa7f072c4a029b23160960b8fc688ca9523ac3"}
May 28 00:36:31.480991 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:31.480572    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerStarted","Data":"e2c850fb21667ed966303d8302f8e83f40d9bee37d98e93b1ec64006046f3066"}
May 28 00:36:31.509750 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:31.509629    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podStartSLOduration=13.509608783000001 podStartE2EDuration="13.509608783s" podCreationTimestamp="2026-05-28 00:36:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:36:31.509560856 +0000 UTC m=+1500.837226967" watchObservedRunningTime="2026-05-28 00:36:31.509608783 +0000 UTC m=+1500.837274893"
May 28 00:36:34.122537 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:34.122483    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:34.123006 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:34.122572    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:34.123006 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:34.122588    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:34.123006 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:34.122634    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:34.124865 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:34.124829    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused"
May 28 00:36:34.124980 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:34.124855    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 28 00:36:34.124980 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:34.124951    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 28 00:36:34.125131 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:34.125096    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused"
May 28 00:36:37.122705 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.122672    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.123166 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.122856    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.123166 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.122980    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.123454 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.123385    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.123454 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.123419    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.123454 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.123434    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.123662 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.123616    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.123714 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.123672    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.123895 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.123879    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.127476 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.127461    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.501478 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.501395    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:36:37.504462 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:36:37.504432    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:37:38.728697 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:38.728663    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"]
May 28 00:37:38.729214 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:38.729053    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="registry" containerID="cri-o://e2c850fb21667ed966303d8302f8e83f40d9bee37d98e93b1ec64006046f3066" gracePeriod=30
May 28 00:37:38.729214 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:38.729191    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="online" containerID="cri-o://778558ea3c1ac9cf603e74e536aa7f072c4a029b23160960b8fc688ca9523ac3" gracePeriod=30
May 28 00:37:38.729326 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:38.729198    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="offline" containerID="cri-o://4a9bd4895a97c3c05ac33b6209036893940c23d65ca3a5cb46594a95b4cdda39" gracePeriod=30
May 28 00:37:38.729375 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:38.729331    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="ui" containerID="cri-o://f6991528a5ff8c82756b9b33755f576631463ead81a6fdaa299a4453a6a24be4" gracePeriod=30
May 28 00:37:39.139819 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.139787    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"]
May 28 00:37:39.143634 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.143613    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.146067 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.146046    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-knbjn\""
May 28 00:37:39.156236 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.156213    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"]
May 28 00:37:39.185503 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.185470    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.185639 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.185549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.185639 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.185575    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.185639 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.185618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.185639 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.185639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ss4l\" (UniqueName: \"kubernetes.io/projected/056c2eee-7462-4e48-9b73-4e325d69a87d-kube-api-access-5ss4l\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.185815 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.185722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/056c2eee-7462-4e48-9b73-4e325d69a87d-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.286551 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.286496    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.286551 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.286549    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5ss4l\" (UniqueName: \"kubernetes.io/projected/056c2eee-7462-4e48-9b73-4e325d69a87d-kube-api-access-5ss4l\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.286777 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.286586    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/056c2eee-7462-4e48-9b73-4e325d69a87d-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.286777 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.286625    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.286777 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.286660    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.286777 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.286685    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.287131 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.287098    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/056c2eee-7462-4e48-9b73-4e325d69a87d-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.289936 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.289905    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.290023 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.289914    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.290023 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.289946    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.290023 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.289978    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.295855 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.295831    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ss4l\" (UniqueName: \"kubernetes.io/projected/056c2eee-7462-4e48-9b73-4e325d69a87d-kube-api-access-5ss4l\") pod \"feast-simple-feast-setup-664fcfcc58-t2hh6\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.454393 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.454366    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:39.582362 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.582332    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"]
May 28 00:37:39.585482 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:37:39.585459    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod056c2eee_7462_4e48_9b73_4e325d69a87d.slice/crio-4b53eb9817c20c8de37a0891675b0234c31581ea298bc0fe0166d139adafec80 WatchSource:0}: Error finding container 4b53eb9817c20c8de37a0891675b0234c31581ea298bc0fe0166d139adafec80: Status 404 returned error can't find the container with id 4b53eb9817c20c8de37a0891675b0234c31581ea298bc0fe0166d139adafec80
May 28 00:37:39.667415 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.667381    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerStarted","Data":"452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03"}
May 28 00:37:39.667539 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.667428    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerStarted","Data":"4b53eb9817c20c8de37a0891675b0234c31581ea298bc0fe0166d139adafec80"}
May 28 00:37:39.670003 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.669981    2560 generic.go:358] "Generic (PLEG): container finished" podID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerID="f6991528a5ff8c82756b9b33755f576631463ead81a6fdaa299a4453a6a24be4" exitCode=0
May 28 00:37:39.670003 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.670001    2560 generic.go:358] "Generic (PLEG): container finished" podID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerID="778558ea3c1ac9cf603e74e536aa7f072c4a029b23160960b8fc688ca9523ac3" exitCode=0
May 28 00:37:39.670137 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.670019    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerDied","Data":"f6991528a5ff8c82756b9b33755f576631463ead81a6fdaa299a4453a6a24be4"}
May 28 00:37:39.670137 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:39.670037    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerDied","Data":"778558ea3c1ac9cf603e74e536aa7f072c4a029b23160960b8fc688ca9523ac3"}
May 28 00:37:44.686566 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:44.686509    2560 generic.go:358] "Generic (PLEG): container finished" podID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerID="452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03" exitCode=0
May 28 00:37:44.686923 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:44.686572    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerDied","Data":"452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03"}
May 28 00:37:45.691337 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:45.691293    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerStarted","Data":"a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01"}
May 28 00:37:47.123657 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:47.123605    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 28 00:37:47.502315 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:47.502208    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 28 00:37:50.709177 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:50.709146    2560 generic.go:358] "Generic (PLEG): container finished" podID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerID="a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01" exitCode=0
May 28 00:37:50.709568 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:50.709187    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerDied","Data":"a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01"}
May 28 00:37:51.717106 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:51.717055    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerStarted","Data":"41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b"}
May 28 00:37:51.717612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:51.717117    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerStarted","Data":"20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863"}
May 28 00:37:51.717612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:51.717132    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerStarted","Data":"1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156"}
May 28 00:37:51.717612 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:51.717147    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerStarted","Data":"f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708"}
May 28 00:37:51.775817 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:51.775750    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podStartSLOduration=12.775730897 podStartE2EDuration="12.775730897s" podCreationTimestamp="2026-05-28 00:37:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:37:51.774030446 +0000 UTC m=+1581.101696555" watchObservedRunningTime="2026-05-28 00:37:51.775730897 +0000 UTC m=+1581.103397005"
May 28 00:37:54.454885 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:54.454845    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:54.454885 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:54.454889    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:54.455429 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:54.454904    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:54.455429 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:54.454917    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:54.456847 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:54.456816    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused"
May 28 00:37:54.457064 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:54.456841    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 28 00:37:54.457158 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:54.456860    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused"
May 28 00:37:54.457158 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:54.456893    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 00:37:57.123835 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.123786    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 28 00:37:57.456092 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.456004    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.456329 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.456310    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.456441 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.456430    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.456846 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.456815    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.456991 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.456853    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.456991 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.456867    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.457082 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.456986    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.457082 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.457045    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.457360 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.457344    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.460730 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.460713    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.502018 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.501985    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 28 00:37:57.741185 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.741103    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:37:57.744243 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:37:57.744220    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:38:07.124033 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:07.123980    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 28 00:38:07.124589 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:07.124129    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:38:07.502359 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:07.502273    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 28 00:38:07.502510 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:07.502387    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:38:08.774081 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:08.774054    2560 generic.go:358] "Generic (PLEG): container finished" podID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerID="4a9bd4895a97c3c05ac33b6209036893940c23d65ca3a5cb46594a95b4cdda39" exitCode=137
May 28 00:38:08.774081 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:08.774079    2560 generic.go:358] "Generic (PLEG): container finished" podID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerID="e2c850fb21667ed966303d8302f8e83f40d9bee37d98e93b1ec64006046f3066" exitCode=137
May 28 00:38:08.774404 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:08.774120    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerDied","Data":"4a9bd4895a97c3c05ac33b6209036893940c23d65ca3a5cb46594a95b4cdda39"}
May 28 00:38:08.774404 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:08.774154    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerDied","Data":"e2c850fb21667ed966303d8302f8e83f40d9bee37d98e93b1ec64006046f3066"}
May 28 00:38:09.387664 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.387642    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:38:09.524825 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.524752    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea76d9a2-64e4-4adc-993b-cf7ef643b557-feast-data\") pod \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") "
May 28 00:38:09.524825 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.524788    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-registry-tls\") pod \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") "
May 28 00:38:09.524825 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.524811    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/ea76d9a2-64e4-4adc-993b-cf7ef643b557-kube-api-access-7k2bl\") pod \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") "
May 28 00:38:09.525075 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.524839    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-ui-tls\") pod \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") "
May 28 00:38:09.525075 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.524857    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-online-tls\") pod \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") "
May 28 00:38:09.525075 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.524884    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-offline-tls\") pod \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\" (UID: \"ea76d9a2-64e4-4adc-993b-cf7ef643b557\") "
May 28 00:38:09.525418 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.525372    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea76d9a2-64e4-4adc-993b-cf7ef643b557-feast-data" (OuterVolumeSpecName: "feast-data") pod "ea76d9a2-64e4-4adc-993b-cf7ef643b557" (UID: "ea76d9a2-64e4-4adc-993b-cf7ef643b557"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:38:09.527220 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.527185    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-online-tls" (OuterVolumeSpecName: "online-tls") pod "ea76d9a2-64e4-4adc-993b-cf7ef643b557" (UID: "ea76d9a2-64e4-4adc-993b-cf7ef643b557"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:38:09.527363 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.527242    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ea76d9a2-64e4-4adc-993b-cf7ef643b557" (UID: "ea76d9a2-64e4-4adc-993b-cf7ef643b557"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:38:09.527363 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.527335    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea76d9a2-64e4-4adc-993b-cf7ef643b557-kube-api-access-7k2bl" (OuterVolumeSpecName: "kube-api-access-7k2bl") pod "ea76d9a2-64e4-4adc-993b-cf7ef643b557" (UID: "ea76d9a2-64e4-4adc-993b-cf7ef643b557"). InnerVolumeSpecName "kube-api-access-7k2bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:38:09.527585 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.527569    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ea76d9a2-64e4-4adc-993b-cf7ef643b557" (UID: "ea76d9a2-64e4-4adc-993b-cf7ef643b557"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:38:09.527646 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.527629    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "ea76d9a2-64e4-4adc-993b-cf7ef643b557" (UID: "ea76d9a2-64e4-4adc-993b-cf7ef643b557"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:38:09.625288 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.625262    2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea76d9a2-64e4-4adc-993b-cf7ef643b557-feast-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:38:09.625288 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.625284    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-registry-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:38:09.625408 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.625293    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/ea76d9a2-64e4-4adc-993b-cf7ef643b557-kube-api-access-7k2bl\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:38:09.625408 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.625303    2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-ui-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:38:09.625408 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.625313    2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-online-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:38:09.625408 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.625322    2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea76d9a2-64e4-4adc-993b-cf7ef643b557-offline-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:38:09.780004 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.779925    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs" event={"ID":"ea76d9a2-64e4-4adc-993b-cf7ef643b557","Type":"ContainerDied","Data":"41a02396491d8462c9343838e0ad4798d523654893e482e4a3abf55be926fa3a"}
May 28 00:38:09.780004 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.779968    2560 scope.go:117] "RemoveContainer" containerID="f6991528a5ff8c82756b9b33755f576631463ead81a6fdaa299a4453a6a24be4"
May 28 00:38:09.780409 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.779966    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"
May 28 00:38:09.788903 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.788886    2560 scope.go:117] "RemoveContainer" containerID="4a9bd4895a97c3c05ac33b6209036893940c23d65ca3a5cb46594a95b4cdda39"
May 28 00:38:09.795793 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.795777    2560 scope.go:117] "RemoveContainer" containerID="778558ea3c1ac9cf603e74e536aa7f072c4a029b23160960b8fc688ca9523ac3"
May 28 00:38:09.802381 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.802323    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"]
May 28 00:38:09.803435 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.803416    2560 scope.go:117] "RemoveContainer" containerID="e2c850fb21667ed966303d8302f8e83f40d9bee37d98e93b1ec64006046f3066"
May 28 00:38:09.804026 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.804001    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-lghgs"]
May 28 00:38:09.810029 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.810013    2560 scope.go:117] "RemoveContainer" containerID="75ba5f63a7e8c526ad69f762c8b0734507ed3e08baba7041d993df44a46748a6"
May 28 00:38:09.816570 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:09.816556    2560 scope.go:117] "RemoveContainer" containerID="4ea30029ee891f2cbf3ce2e4436e814d22cbfc385f15c1600c4a964ceea6f61c"
May 28 00:38:11.261388 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:38:11.261349    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" path="/var/lib/kubelet/pods/ea76d9a2-64e4-4adc-993b-cf7ef643b557/volumes"
May 28 00:39:00.411556 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.411500    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"]
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.411913    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="online"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.411931    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="online"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.411948    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="registry"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.411956    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="registry"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.411967    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="feast-init"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.411976    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="feast-init"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.411990    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="feast-apply"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.411998    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="feast-apply"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.412022    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="offline"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.412030    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="offline"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.412042    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="ui"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.412050    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="ui"
May 28 00:39:00.412117 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.412116    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="ui"
May 28 00:39:00.412796 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.412127    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="registry"
May 28 00:39:00.412796 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.412138    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="online"
May 28 00:39:00.412796 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.412147    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea76d9a2-64e4-4adc-993b-cf7ef643b557" containerName="offline"
May 28 00:39:00.415680 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.415659    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.417732 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.417709    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 00:39:00.417863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.417838    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 00:39:00.417863 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.417851    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-pchg2\""
May 28 00:39:00.417994 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.417840    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 00:39:00.417994 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.417857    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 00:39:00.418088 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.418014    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 00:39:00.418194 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.418181    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 00:39:00.427267 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.427249    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"]
May 28 00:39:00.493529 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.493485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5nn\" (UniqueName: \"kubernetes.io/projected/2a197df8-9c54-442d-b801-3916cc4cd4a7-kube-api-access-lh5nn\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.493661 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.493543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.493661 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.493622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.493767 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.493676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a197df8-9c54-442d-b801-3916cc4cd4a7-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.493767 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.493707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a197df8-9c54-442d-b801-3916cc4cd4a7-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.493834 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.493769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.594542 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.594492    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5nn\" (UniqueName: \"kubernetes.io/projected/2a197df8-9c54-442d-b801-3916cc4cd4a7-kube-api-access-lh5nn\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.594671 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.594561    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.594671 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.594596    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.594671 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.594619    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a197df8-9c54-442d-b801-3916cc4cd4a7-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.594671 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.594644    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a197df8-9c54-442d-b801-3916cc4cd4a7-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.594883 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:39:00.594690    2560 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:39:00.594883 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:39:00.594723    2560 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 00:39:00.594883 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:39:00.594767    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-online-tls podName:2a197df8-9c54-442d-b801-3916cc4cd4a7 nodeName:}" failed. No retries permitted until 2026-05-28 00:39:01.094745569 +0000 UTC m=+1650.422411656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-online-tls") pod "feast-simple-feast-remote-setup-5cff94746b-9rdc2" (UID: "2a197df8-9c54-442d-b801-3916cc4cd4a7") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:39:00.594883 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.594689    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.594883 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:39:00.594793    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-ui-tls podName:2a197df8-9c54-442d-b801-3916cc4cd4a7 nodeName:}" failed. No retries permitted until 2026-05-28 00:39:01.094775297 +0000 UTC m=+1650.422441386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-ui-tls") pod "feast-simple-feast-remote-setup-5cff94746b-9rdc2" (UID: "2a197df8-9c54-442d-b801-3916cc4cd4a7") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 00:39:00.595101 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.595058    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a197df8-9c54-442d-b801-3916cc4cd4a7-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.595353 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.595335    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a197df8-9c54-442d-b801-3916cc4cd4a7-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.596965 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.596947    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:00.604180 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:00.604160    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5nn\" (UniqueName: \"kubernetes.io/projected/2a197df8-9c54-442d-b801-3916cc4cd4a7-kube-api-access-lh5nn\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:01.098486 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:01.098453    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:01.098703 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:01.098583    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:01.100976 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:01.100949    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:01.101101 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:01.101058    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-9rdc2\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:01.325504 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:01.325466    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:01.456010 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:01.455985    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"]
May 28 00:39:01.458361 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:39:01.458333    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a197df8_9c54_442d_b801_3916cc4cd4a7.slice/crio-1daa2ad55ee2a424fe76b95a75532be9b76693011f73805b79d2cef5f0dd454d WatchSource:0}: Error finding container 1daa2ad55ee2a424fe76b95a75532be9b76693011f73805b79d2cef5f0dd454d: Status 404 returned error can't find the container with id 1daa2ad55ee2a424fe76b95a75532be9b76693011f73805b79d2cef5f0dd454d
May 28 00:39:01.923945 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:01.923905    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerStarted","Data":"a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426"}
May 28 00:39:01.924135 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:01.923956    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerStarted","Data":"1daa2ad55ee2a424fe76b95a75532be9b76693011f73805b79d2cef5f0dd454d"}
May 28 00:39:06.940475 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:06.940442    2560 generic.go:358] "Generic (PLEG): container finished" podID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerID="a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426" exitCode=0
May 28 00:39:06.940871 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:06.940479    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerDied","Data":"a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426"}
May 28 00:39:07.945241 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:07.945202    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerStarted","Data":"112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a"}
May 28 00:39:12.959913 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:12.959875    2560 generic.go:358] "Generic (PLEG): container finished" podID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerID="112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a" exitCode=0
May 28 00:39:12.960272 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:12.959952    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerDied","Data":"112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a"}
May 28 00:39:13.966410 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:13.966370    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerStarted","Data":"d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7"}
May 28 00:39:13.966410 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:13.966415    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerStarted","Data":"4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581"}
May 28 00:39:13.966948 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:13.966425    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerStarted","Data":"c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205"}
May 28 00:39:13.990574 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:13.990487    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podStartSLOduration=13.990468447 podStartE2EDuration="13.990468447s" podCreationTimestamp="2026-05-28 00:39:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:39:13.989972553 +0000 UTC m=+1663.317638706" watchObservedRunningTime="2026-05-28 00:39:13.990468447 +0000 UTC m=+1663.318134556"
May 28 00:39:16.325870 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:16.325828    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:16.325870 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:16.325874    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:16.326386 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:16.325888    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:16.327158 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:16.327112    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 28 00:39:16.327158 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:16.327125    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused"
May 28 00:39:16.327333 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:16.327130    2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 00:39:19.326822 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:19.326785    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:19.327268 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:19.326923    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:19.327373 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:19.327344    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:19.327504 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:19.327380    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:19.327677 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:19.327657    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:19.327738 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:19.327716    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:19.331383 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:19.331363    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:19.984954 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:19.984922    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:39:19.987825 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:39:19.987801    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:40:21.414199 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:21.414163    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"]
May 28 00:40:21.414794 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:21.414597    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="online" containerID="cri-o://c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205" gracePeriod=30
May 28 00:40:21.414794 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:21.414732    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="offline" containerID="cri-o://4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581" gracePeriod=30
May 28 00:40:21.414926 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:21.414878    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="ui" containerID="cri-o://d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7" gracePeriod=30
May 28 00:40:21.569278 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:21.569240    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"]
May 28 00:40:21.569738 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:21.569689    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="registry" containerID="cri-o://f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708" gracePeriod=30
May 28 00:40:21.569872 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:21.569850    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="ui" containerID="cri-o://41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b" gracePeriod=30
May 28 00:40:21.569946 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:21.569929    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="offline" containerID="cri-o://20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863" gracePeriod=30
May 28 00:40:21.570007 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:21.569991    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="online" containerID="cri-o://1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156" gracePeriod=30
May 28 00:40:22.167191 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:22.167106    2560 generic.go:358] "Generic (PLEG): container finished" podID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerID="41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b" exitCode=0
May 28 00:40:22.167191 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:22.167174    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerDied","Data":"41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b"}
May 28 00:40:22.169413 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:22.169389    2560 generic.go:358] "Generic (PLEG): container finished" podID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerID="d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7" exitCode=0
May 28 00:40:22.169413 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:22.169407    2560 generic.go:358] "Generic (PLEG): container finished" podID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerID="c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205" exitCode=0
May 28 00:40:22.169620 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:22.169463    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerDied","Data":"d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7"}
May 28 00:40:22.169620 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:22.169494    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerDied","Data":"c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205"}
May 28 00:40:23.174326 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:23.174291    2560 generic.go:358] "Generic (PLEG): container finished" podID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerID="1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156" exitCode=0
May 28 00:40:23.174686 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:23.174334    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerDied","Data":"1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156"}
May 28 00:40:27.457089 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:27.457044    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 28 00:40:27.742032 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:27.741941    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 00:40:29.327653 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:29.327611    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 28 00:40:29.985979 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:29.985931    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 00:40:37.457102 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:37.457046    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 28 00:40:37.741475 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:37.741383    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 00:40:39.328051 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:39.328012    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 28 00:40:39.985168 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:39.985118    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 00:40:47.457330 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:47.457278    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 28 00:40:47.457801 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:47.457430    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:40:47.742232 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:47.742145    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 00:40:47.742369 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:47.742256    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:40:49.328411 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:49.328366    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 28 00:40:49.328815 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:49.328480    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:40:49.985675 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:49.985640    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 00:40:49.985849 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:49.985745    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:40:51.598837 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.598815    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:40:51.612485 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.612462    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a197df8-9c54-442d-b801-3916cc4cd4a7-feast-data\") pod \"2a197df8-9c54-442d-b801-3916cc4cd4a7\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") "
May 28 00:40:51.612671 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.612503    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh5nn\" (UniqueName: \"kubernetes.io/projected/2a197df8-9c54-442d-b801-3916cc4cd4a7-kube-api-access-lh5nn\") pod \"2a197df8-9c54-442d-b801-3916cc4cd4a7\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") "
May 28 00:40:51.612671 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.612564    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-ui-tls\") pod \"2a197df8-9c54-442d-b801-3916cc4cd4a7\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") "
May 28 00:40:51.612671 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.612620    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-offline-tls\") pod \"2a197df8-9c54-442d-b801-3916cc4cd4a7\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") "
May 28 00:40:51.612671 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.612666    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-online-tls\") pod \"2a197df8-9c54-442d-b801-3916cc4cd4a7\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") "
May 28 00:40:51.612876 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.612743    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a197df8-9c54-442d-b801-3916cc4cd4a7-registry-tls\") pod \"2a197df8-9c54-442d-b801-3916cc4cd4a7\" (UID: \"2a197df8-9c54-442d-b801-3916cc4cd4a7\") "
May 28 00:40:51.613226 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.612917    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a197df8-9c54-442d-b801-3916cc4cd4a7-feast-data" (OuterVolumeSpecName: "feast-data") pod "2a197df8-9c54-442d-b801-3916cc4cd4a7" (UID: "2a197df8-9c54-442d-b801-3916cc4cd4a7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:40:51.613306 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.613246    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a197df8-9c54-442d-b801-3916cc4cd4a7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2a197df8-9c54-442d-b801-3916cc4cd4a7" (UID: "2a197df8-9c54-442d-b801-3916cc4cd4a7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:40:51.654070 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.654041    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a197df8-9c54-442d-b801-3916cc4cd4a7-kube-api-access-lh5nn" (OuterVolumeSpecName: "kube-api-access-lh5nn") pod "2a197df8-9c54-442d-b801-3916cc4cd4a7" (UID: "2a197df8-9c54-442d-b801-3916cc4cd4a7"). InnerVolumeSpecName "kube-api-access-lh5nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:40:51.654070 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.654055    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2a197df8-9c54-442d-b801-3916cc4cd4a7" (UID: "2a197df8-9c54-442d-b801-3916cc4cd4a7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:40:51.654248 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.654073    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-online-tls" (OuterVolumeSpecName: "online-tls") pod "2a197df8-9c54-442d-b801-3916cc4cd4a7" (UID: "2a197df8-9c54-442d-b801-3916cc4cd4a7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:40:51.654248 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.654091    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2a197df8-9c54-442d-b801-3916cc4cd4a7" (UID: "2a197df8-9c54-442d-b801-3916cc4cd4a7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:40:51.713607 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.713533    2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a197df8-9c54-442d-b801-3916cc4cd4a7-feast-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:51.713607 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.713569    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5nn\" (UniqueName: \"kubernetes.io/projected/2a197df8-9c54-442d-b801-3916cc4cd4a7-kube-api-access-lh5nn\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:51.713607 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.713581    2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-ui-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:51.713607 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.713590    2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-offline-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:51.713607 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.713599    2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a197df8-9c54-442d-b801-3916cc4cd4a7-online-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:51.713607 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:51.713607    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a197df8-9c54-442d-b801-3916cc4cd4a7-registry-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:52.214120 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.214099    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:40:52.254647 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.254613    2560 generic.go:358] "Generic (PLEG): container finished" podID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerID="20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863" exitCode=137
May 28 00:40:52.254647 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.254638    2560 generic.go:358] "Generic (PLEG): container finished" podID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerID="f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708" exitCode=137
May 28 00:40:52.254856 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.254701    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerDied","Data":"20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863"}
May 28 00:40:52.254856 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.254709    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"
May 28 00:40:52.254856 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.254741    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerDied","Data":"f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708"}
May 28 00:40:52.254856 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.254755    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6" event={"ID":"056c2eee-7462-4e48-9b73-4e325d69a87d","Type":"ContainerDied","Data":"4b53eb9817c20c8de37a0891675b0234c31581ea298bc0fe0166d139adafec80"}
May 28 00:40:52.254856 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.254777    2560 scope.go:117] "RemoveContainer" containerID="41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b"
May 28 00:40:52.257194 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.257171    2560 generic.go:358] "Generic (PLEG): container finished" podID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerID="4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581" exitCode=137
May 28 00:40:52.257283 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.257203    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerDied","Data":"4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581"}
May 28 00:40:52.257283 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.257226    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2" event={"ID":"2a197df8-9c54-442d-b801-3916cc4cd4a7","Type":"ContainerDied","Data":"1daa2ad55ee2a424fe76b95a75532be9b76693011f73805b79d2cef5f0dd454d"}
May 28 00:40:52.257283 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.257238    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"
May 28 00:40:52.263711 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.263695    2560 scope.go:117] "RemoveContainer" containerID="20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863"
May 28 00:40:52.271228 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.271212    2560 scope.go:117] "RemoveContainer" containerID="1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156"
May 28 00:40:52.277874 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.277857    2560 scope.go:117] "RemoveContainer" containerID="f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708"
May 28 00:40:52.281071 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.281051    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"]
May 28 00:40:52.286377 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.286360    2560 scope.go:117] "RemoveContainer" containerID="a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01"
May 28 00:40:52.286547 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.286506    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-9rdc2"]
May 28 00:40:52.292864 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.292848    2560 scope.go:117] "RemoveContainer" containerID="452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03"
May 28 00:40:52.299470 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.299454    2560 scope.go:117] "RemoveContainer" containerID="41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b"
May 28 00:40:52.299778 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.299759    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b\": container with ID starting with 41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b not found: ID does not exist" containerID="41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b"
May 28 00:40:52.299859 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.299788    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b"} err="failed to get container status \"41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b\": rpc error: code = NotFound desc = could not find container \"41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b\": container with ID starting with 41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b not found: ID does not exist"
May 28 00:40:52.299859 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.299812    2560 scope.go:117] "RemoveContainer" containerID="20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863"
May 28 00:40:52.300069 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.300050    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863\": container with ID starting with 20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863 not found: ID does not exist" containerID="20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863"
May 28 00:40:52.300110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.300076    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863"} err="failed to get container status \"20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863\": rpc error: code = NotFound desc = could not find container \"20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863\": container with ID starting with 20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863 not found: ID does not exist"
May 28 00:40:52.300110 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.300094    2560 scope.go:117] "RemoveContainer" containerID="1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156"
May 28 00:40:52.300354 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.300337    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156\": container with ID starting with 1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156 not found: ID does not exist" containerID="1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156"
May 28 00:40:52.300393 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.300359    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156"} err="failed to get container status \"1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156\": rpc error: code = NotFound desc = could not find container \"1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156\": container with ID starting with 1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156 not found: ID does not exist"
May 28 00:40:52.300393 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.300375    2560 scope.go:117] "RemoveContainer" containerID="f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708"
May 28 00:40:52.300659 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.300641    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708\": container with ID starting with f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708 not found: ID does not exist" containerID="f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708"
May 28 00:40:52.300733 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.300667    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708"} err="failed to get container status \"f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708\": rpc error: code = NotFound desc = could not find container \"f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708\": container with ID starting with f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708 not found: ID does not exist"
May 28 00:40:52.300733 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.300687    2560 scope.go:117] "RemoveContainer" containerID="a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01"
May 28 00:40:52.300926 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.300908    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01\": container with ID starting with a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01 not found: ID does not exist" containerID="a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01"
May 28 00:40:52.300992 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.300932    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01"} err="failed to get container status \"a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01\": rpc error: code = NotFound desc = could not find container \"a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01\": container with ID starting with a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01 not found: ID does not exist"
May 28 00:40:52.300992 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.300952    2560 scope.go:117] "RemoveContainer" containerID="452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03"
May 28 00:40:52.301165 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.301148    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03\": container with ID starting with 452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03 not found: ID does not exist" containerID="452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03"
May 28 00:40:52.301212 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.301174    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03"} err="failed to get container status \"452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03\": rpc error: code = NotFound desc = could not find container \"452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03\": container with ID starting with 452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03 not found: ID does not exist"
May 28 00:40:52.301212 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.301190    2560 scope.go:117] "RemoveContainer" containerID="41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b"
May 28 00:40:52.301432 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.301417    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b"} err="failed to get container status \"41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b\": rpc error: code = NotFound desc = could not find container \"41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b\": container with ID starting with 41c5d1a97670e9756bb2bc50227fa79fc7c6144193835ed6a9617c402b614b8b not found: ID does not exist"
May 28 00:40:52.301467 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.301433    2560 scope.go:117] "RemoveContainer" containerID="20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863"
May 28 00:40:52.301688 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.301669    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863"} err="failed to get container status \"20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863\": rpc error: code = NotFound desc = could not find container \"20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863\": container with ID starting with 20f08469135d4c0747319dbed528de4198bbfe1dbd77ca281d7b644edb732863 not found: ID does not exist"
May 28 00:40:52.301760 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.301689    2560 scope.go:117] "RemoveContainer" containerID="1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156"
May 28 00:40:52.301901 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.301883    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156"} err="failed to get container status \"1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156\": rpc error: code = NotFound desc = could not find container \"1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156\": container with ID starting with 1dd91ec662bfb5974dc425f80946d7948fbb09c79a399bb1f86c84a8293a6156 not found: ID does not exist"
May 28 00:40:52.301901 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.301901    2560 scope.go:117] "RemoveContainer" containerID="f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708"
May 28 00:40:52.302108 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.302086    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708"} err="failed to get container status \"f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708\": rpc error: code = NotFound desc = could not find container \"f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708\": container with ID starting with f954fd2aef88dbf2cadbf4c9220f5d7a9aeac25c8402988bbdfb7cad132f8708 not found: ID does not exist"
May 28 00:40:52.302147 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.302109    2560 scope.go:117] "RemoveContainer" containerID="a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01"
May 28 00:40:52.302319 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.302303    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01"} err="failed to get container status \"a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01\": rpc error: code = NotFound desc = could not find container \"a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01\": container with ID starting with a3e55e35a2c0f1e05e5be0b30f793491d7531aee2e2aecf45a2f851adfd5db01 not found: ID does not exist"
May 28 00:40:52.302370 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.302320    2560 scope.go:117] "RemoveContainer" containerID="452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03"
May 28 00:40:52.302560 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.302542    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03"} err="failed to get container status \"452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03\": rpc error: code = NotFound desc = could not find container \"452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03\": container with ID starting with 452f3495cb0657068c1a0bf7d7213189b7d1de6cd1d9ce60a91cc6fe38d3be03 not found: ID does not exist"
May 28 00:40:52.302608 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.302562    2560 scope.go:117] "RemoveContainer" containerID="d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7"
May 28 00:40:52.309441 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.309425    2560 scope.go:117] "RemoveContainer" containerID="4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581"
May 28 00:40:52.315709 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.315695    2560 scope.go:117] "RemoveContainer" containerID="c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205"
May 28 00:40:52.317789 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.317769    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-registry-tls\") pod \"056c2eee-7462-4e48-9b73-4e325d69a87d\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") "
May 28 00:40:52.317869 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.317811    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/056c2eee-7462-4e48-9b73-4e325d69a87d-feast-data\") pod \"056c2eee-7462-4e48-9b73-4e325d69a87d\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") "
May 28 00:40:52.317917 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.317897    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-ui-tls\") pod \"056c2eee-7462-4e48-9b73-4e325d69a87d\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") "
May 28 00:40:52.317952 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.317932    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ss4l\" (UniqueName: \"kubernetes.io/projected/056c2eee-7462-4e48-9b73-4e325d69a87d-kube-api-access-5ss4l\") pod \"056c2eee-7462-4e48-9b73-4e325d69a87d\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") "
May 28 00:40:52.318004 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.317959    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-offline-tls\") pod \"056c2eee-7462-4e48-9b73-4e325d69a87d\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") "
May 28 00:40:52.318004 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.317987    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-online-tls\") pod \"056c2eee-7462-4e48-9b73-4e325d69a87d\" (UID: \"056c2eee-7462-4e48-9b73-4e325d69a87d\") "
May 28 00:40:52.318366 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.318334    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056c2eee-7462-4e48-9b73-4e325d69a87d-feast-data" (OuterVolumeSpecName: "feast-data") pod "056c2eee-7462-4e48-9b73-4e325d69a87d" (UID: "056c2eee-7462-4e48-9b73-4e325d69a87d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:40:52.320338 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.320301    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056c2eee-7462-4e48-9b73-4e325d69a87d-kube-api-access-5ss4l" (OuterVolumeSpecName: "kube-api-access-5ss4l") pod "056c2eee-7462-4e48-9b73-4e325d69a87d" (UID: "056c2eee-7462-4e48-9b73-4e325d69a87d"). InnerVolumeSpecName "kube-api-access-5ss4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:40:52.320709 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.320687    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "056c2eee-7462-4e48-9b73-4e325d69a87d" (UID: "056c2eee-7462-4e48-9b73-4e325d69a87d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:40:52.320797 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.320776    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-online-tls" (OuterVolumeSpecName: "online-tls") pod "056c2eee-7462-4e48-9b73-4e325d69a87d" (UID: "056c2eee-7462-4e48-9b73-4e325d69a87d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:40:52.320848 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.320819    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "056c2eee-7462-4e48-9b73-4e325d69a87d" (UID: "056c2eee-7462-4e48-9b73-4e325d69a87d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:40:52.320883 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.320860    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "056c2eee-7462-4e48-9b73-4e325d69a87d" (UID: "056c2eee-7462-4e48-9b73-4e325d69a87d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:40:52.322855 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.322833    2560 scope.go:117] "RemoveContainer" containerID="112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a"
May 28 00:40:52.331195 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.331167    2560 scope.go:117] "RemoveContainer" containerID="a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426"
May 28 00:40:52.339340 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.339319    2560 scope.go:117] "RemoveContainer" containerID="d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7"
May 28 00:40:52.339658 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.339629    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7\": container with ID starting with d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7 not found: ID does not exist" containerID="d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7"
May 28 00:40:52.339800 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.339677    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7"} err="failed to get container status \"d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7\": rpc error: code = NotFound desc = could not find container \"d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7\": container with ID starting with d1d0f9a743ce15e8f80a4b8eab71df21ac925af865a99aba96034ddaa778e8e7 not found: ID does not exist"
May 28 00:40:52.339800 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.339699    2560 scope.go:117] "RemoveContainer" containerID="4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581"
May 28 00:40:52.339963 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.339944    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581\": container with ID starting with 4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581 not found: ID does not exist" containerID="4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581"
May 28 00:40:52.340019 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.339975    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581"} err="failed to get container status \"4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581\": rpc error: code = NotFound desc = could not find container \"4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581\": container with ID starting with 4bb46b33262dd90e2a2472111a8447b9253c9ba17c31b2db3a7d424781086581 not found: ID does not exist"
May 28 00:40:52.340019 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.339994    2560 scope.go:117] "RemoveContainer" containerID="c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205"
May 28 00:40:52.340282 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.340259    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205\": container with ID starting with c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205 not found: ID does not exist" containerID="c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205"
May 28 00:40:52.340353 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.340289    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205"} err="failed to get container status \"c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205\": rpc error: code = NotFound desc = could not find container \"c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205\": container with ID starting with c269ec9f8e1672cf5cc2b278be792b7aa2dcf33156aec633717a9d780ab98205 not found: ID does not exist"
May 28 00:40:52.340353 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.340307    2560 scope.go:117] "RemoveContainer" containerID="112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a"
May 28 00:40:52.340648 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.340630    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a\": container with ID starting with 112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a not found: ID does not exist" containerID="112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a"
May 28 00:40:52.340728 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.340651    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a"} err="failed to get container status \"112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a\": rpc error: code = NotFound desc = could not find container \"112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a\": container with ID starting with 112cfc8f9fdd9446763d66984173bdd0e888bee0ab0faf70a8856174e6579c4a not found: ID does not exist"
May 28 00:40:52.340728 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.340665    2560 scope.go:117] "RemoveContainer" containerID="a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426"
May 28 00:40:52.340901 ip-10-0-142-230 kubenswrapper[2560]: E0528 00:40:52.340885    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426\": container with ID starting with a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426 not found: ID does not exist" containerID="a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426"
May 28 00:40:52.340961 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.340903    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426"} err="failed to get container status \"a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426\": rpc error: code = NotFound desc = could not find container \"a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426\": container with ID starting with a4cac987e03cab4816c19c20d10e13a6e03a544eaf9e41c02f57cf949164d426 not found: ID does not exist"
May 28 00:40:52.419265 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.419244    2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-online-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:52.419265 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.419266    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-registry-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:52.419452 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.419275    2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/056c2eee-7462-4e48-9b73-4e325d69a87d-feast-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:52.419452 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.419283    2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-ui-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:52.419452 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.419291    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ss4l\" (UniqueName: \"kubernetes.io/projected/056c2eee-7462-4e48-9b73-4e325d69a87d-kube-api-access-5ss4l\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:52.419452 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.419299    2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/056c2eee-7462-4e48-9b73-4e325d69a87d-offline-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
May 28 00:40:52.577112 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.577074    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"]
May 28 00:40:52.580705 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:52.580683    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-t2hh6"]
May 28 00:40:53.260803 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:53.260765    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" path="/var/lib/kubelet/pods/056c2eee-7462-4e48-9b73-4e325d69a87d/volumes"
May 28 00:40:53.261552 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:40:53.261508    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" path="/var/lib/kubelet/pods/2a197df8-9c54-442d-b801-3916cc4cd4a7/volumes"
May 28 00:41:28.428968 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.428925    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nc6j4/must-gather-fjkh7"]
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429182    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="feast-apply"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429194    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="feast-apply"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429206    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="feast-init"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429218    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="feast-init"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429228    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="registry"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429234    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="registry"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429240    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="offline"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429246    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="offline"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429254    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="feast-init"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429260    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="feast-init"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429266    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="online"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429271    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="online"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429278    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="offline"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429283    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="offline"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429289    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="feast-apply"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429294    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="feast-apply"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429301    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="ui"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429306    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="ui"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429312    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="ui"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429317    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="ui"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429324    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="online"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429329    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="online"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429365    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="online"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429373    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="offline"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429380    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="offline"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429385    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="registry"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429391    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="online"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429397    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="056c2eee-7462-4e48-9b73-4e325d69a87d" containerName="ui"
May 28 00:41:28.429488 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.429403    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a197df8-9c54-442d-b801-3916cc4cd4a7" containerName="ui"
May 28 00:41:28.432297 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.432281    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6j4/must-gather-fjkh7"
May 28 00:41:28.434463 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.434440    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6j4\"/\"kube-root-ca.crt\""
May 28 00:41:28.434463 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.434440    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nc6j4\"/\"default-dockercfg-hwxt8\""
May 28 00:41:28.434645 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.434483    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6j4\"/\"openshift-service-ca.crt\""
May 28 00:41:28.438283 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.438261    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6j4/must-gather-fjkh7"]
May 28 00:41:28.575029 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.575001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6v9x\" (UniqueName: \"kubernetes.io/projected/b41f837e-efd0-41d2-9ebd-bae6e31d871f-kube-api-access-c6v9x\") pod \"must-gather-fjkh7\" (UID: \"b41f837e-efd0-41d2-9ebd-bae6e31d871f\") " pod="openshift-must-gather-nc6j4/must-gather-fjkh7"
May 28 00:41:28.575199 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.575044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b41f837e-efd0-41d2-9ebd-bae6e31d871f-must-gather-output\") pod \"must-gather-fjkh7\" (UID: \"b41f837e-efd0-41d2-9ebd-bae6e31d871f\") " pod="openshift-must-gather-nc6j4/must-gather-fjkh7"
May 28 00:41:28.676349 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.676314    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c6v9x\" (UniqueName: \"kubernetes.io/projected/b41f837e-efd0-41d2-9ebd-bae6e31d871f-kube-api-access-c6v9x\") pod \"must-gather-fjkh7\" (UID: \"b41f837e-efd0-41d2-9ebd-bae6e31d871f\") " pod="openshift-must-gather-nc6j4/must-gather-fjkh7"
May 28 00:41:28.676349 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.676355    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b41f837e-efd0-41d2-9ebd-bae6e31d871f-must-gather-output\") pod \"must-gather-fjkh7\" (UID: \"b41f837e-efd0-41d2-9ebd-bae6e31d871f\") " pod="openshift-must-gather-nc6j4/must-gather-fjkh7"
May 28 00:41:28.676695 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.676677    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b41f837e-efd0-41d2-9ebd-bae6e31d871f-must-gather-output\") pod \"must-gather-fjkh7\" (UID: \"b41f837e-efd0-41d2-9ebd-bae6e31d871f\") " pod="openshift-must-gather-nc6j4/must-gather-fjkh7"
May 28 00:41:28.684392 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.684347    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6v9x\" (UniqueName: \"kubernetes.io/projected/b41f837e-efd0-41d2-9ebd-bae6e31d871f-kube-api-access-c6v9x\") pod \"must-gather-fjkh7\" (UID: \"b41f837e-efd0-41d2-9ebd-bae6e31d871f\") " pod="openshift-must-gather-nc6j4/must-gather-fjkh7"
May 28 00:41:28.741607 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.741573    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6j4/must-gather-fjkh7"
May 28 00:41:28.865314 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.865280    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6j4/must-gather-fjkh7"]
May 28 00:41:28.868125 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:41:28.868099    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb41f837e_efd0_41d2_9ebd_bae6e31d871f.slice/crio-642eca274d30de622acb53459346012a36dd401b824e856b71f533735818826d WatchSource:0}: Error finding container 642eca274d30de622acb53459346012a36dd401b824e856b71f533735818826d: Status 404 returned error can't find the container with id 642eca274d30de622acb53459346012a36dd401b824e856b71f533735818826d
May 28 00:41:28.869933 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:28.869908    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:41:29.350710 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:29.350677    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6j4/must-gather-fjkh7" event={"ID":"b41f837e-efd0-41d2-9ebd-bae6e31d871f","Type":"ContainerStarted","Data":"642eca274d30de622acb53459346012a36dd401b824e856b71f533735818826d"}
May 28 00:41:30.355074 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:30.355032    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6j4/must-gather-fjkh7" event={"ID":"b41f837e-efd0-41d2-9ebd-bae6e31d871f","Type":"ContainerStarted","Data":"d110f2cdeb14dd2bca244128997e38d4910d908420292d28cf8a63d9abf7a460"}
May 28 00:41:30.355074 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:30.355080    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6j4/must-gather-fjkh7" event={"ID":"b41f837e-efd0-41d2-9ebd-bae6e31d871f","Type":"ContainerStarted","Data":"3238f798e5a5b26b1f8eee2748454f46721827f82d87e330a41d2a3f1d9ee845"}
May 28 00:41:31.227695 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:31.227667    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-87tk8_57d0ed4b-1a8c-4e6f-80ea-a56a3a18d112/global-pull-secret-syncer/0.log"
May 28 00:41:31.389334 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:31.389267    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-45gzz_606fba4b-6867-42fe-b598-0dc46cf45c5b/konnectivity-agent/0.log"
May 28 00:41:31.525459 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:31.525384    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-230.ec2.internal_6fac2f13116ec505f7f49d6272df8583/haproxy/0.log"
May 28 00:41:34.406598 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:34.406559    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-fsmz8_b5068452-b413-411d-a928-b8e91a4b942a/cluster-monitoring-operator/0.log"
May 28 00:41:34.536700 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:34.536668    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-vdchp_0d360f35-f6bc-43d7-a34e-d8175e7b7a52/monitoring-plugin/0.log"
May 28 00:41:34.750079 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:34.749997    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rghgw_6819b656-c4e6-44c5-890d-a52c556e714a/node-exporter/0.log"
May 28 00:41:34.773852 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:34.773820    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rghgw_6819b656-c4e6-44c5-890d-a52c556e714a/kube-rbac-proxy/0.log"
May 28 00:41:34.799259 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:34.799232    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rghgw_6819b656-c4e6-44c5-890d-a52c556e714a/init-textfile/0.log"
May 28 00:41:37.267599 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.267568    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-mtcq5_2d6f4edf-5055-40c8-a687-08f1753d6bc0/download-server/0.log"
May 28 00:41:37.822312 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.822252    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-nc6j4/must-gather-fjkh7" podStartSLOduration=8.811899123 podStartE2EDuration="9.822233349s" podCreationTimestamp="2026-05-28 00:41:28 +0000 UTC" firstStartedPulling="2026-05-28 00:41:28.870063635 +0000 UTC m=+1798.197729721" lastFinishedPulling="2026-05-28 00:41:29.880397846 +0000 UTC m=+1799.208063947" observedRunningTime="2026-05-28 00:41:30.3750329 +0000 UTC m=+1799.702699007" watchObservedRunningTime="2026-05-28 00:41:37.822233349 +0000 UTC m=+1807.149899458"
May 28 00:41:37.823700 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.823671    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"]
May 28 00:41:37.827116 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.827088    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:37.839100 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.839067    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"]
May 28 00:41:37.960352 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.960321    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-lib-modules\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:37.960352 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.960359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-sys\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:37.960622 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.960383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-podres\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:37.960622 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.960434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcj2l\" (UniqueName: \"kubernetes.io/projected/da3c11a6-ef54-4f53-a11d-f6a40f129461-kube-api-access-gcj2l\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:37.960622 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:37.960483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-proc\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.061414 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.061382    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-lib-modules\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.061606 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.061420    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-sys\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.061606 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.061444    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-podres\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.061606 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.061476    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gcj2l\" (UniqueName: \"kubernetes.io/projected/da3c11a6-ef54-4f53-a11d-f6a40f129461-kube-api-access-gcj2l\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.061606 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.061539    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-sys\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.061606 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.061553    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-proc\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.061606 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.061584    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-lib-modules\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.061827 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.061624    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-proc\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.061827 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.061642    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da3c11a6-ef54-4f53-a11d-f6a40f129461-podres\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.069439 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.069417    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcj2l\" (UniqueName: \"kubernetes.io/projected/da3c11a6-ef54-4f53-a11d-f6a40f129461-kube-api-access-gcj2l\") pod \"perf-node-gather-daemonset-prqzq\" (UID: \"da3c11a6-ef54-4f53-a11d-f6a40f129461\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.137290 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.137237    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.289983 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.289950    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"]
May 28 00:41:38.293856 ip-10-0-142-230 kubenswrapper[2560]: W0528 00:41:38.293822    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podda3c11a6_ef54_4f53_a11d_f6a40f129461.slice/crio-fa974645b34ad42002219a7598071626e76f51b600e016a227cdcd71916f0420 WatchSource:0}: Error finding container fa974645b34ad42002219a7598071626e76f51b600e016a227cdcd71916f0420: Status 404 returned error can't find the container with id fa974645b34ad42002219a7598071626e76f51b600e016a227cdcd71916f0420
May 28 00:41:38.352704 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.352682    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nkppb_200a652c-a7c2-4aee-8767-26801182f407/dns/0.log"
May 28 00:41:38.374962 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.374941    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nkppb_200a652c-a7c2-4aee-8767-26801182f407/kube-rbac-proxy/0.log"
May 28 00:41:38.389600 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.389547    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq" event={"ID":"da3c11a6-ef54-4f53-a11d-f6a40f129461","Type":"ContainerStarted","Data":"0745349b4af62f1cb0c324f8bf184457a12e084f443e8ce5b9fc3fb4dba03be4"}
May 28 00:41:38.389600 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.389587    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq" event={"ID":"da3c11a6-ef54-4f53-a11d-f6a40f129461","Type":"ContainerStarted","Data":"fa974645b34ad42002219a7598071626e76f51b600e016a227cdcd71916f0420"}
May 28 00:41:38.389736 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.389622    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:38.406636 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.406600    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq" podStartSLOduration=1.406586736 podStartE2EDuration="1.406586736s" podCreationTimestamp="2026-05-28 00:41:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:41:38.406020844 +0000 UTC m=+1807.733686952" watchObservedRunningTime="2026-05-28 00:41:38.406586736 +0000 UTC m=+1807.734252844"
May 28 00:41:38.452199 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.452181    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7qxzh_bbde4ba0-8948-4141-93a0-c0ff1aa62641/dns-node-resolver/0.log"
May 28 00:41:38.876626 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.876587    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-577655c79f-9zmnl_66068cd5-441a-4aac-819d-47ededa2a63f/registry/0.log"
May 28 00:41:38.955400 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:38.955367    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qfqhg_9315e6c2-c11e-4ee7-a5fd-5f57e12772a6/node-ca/0.log"
May 28 00:41:39.606223 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:39.606190    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74d5c8c648-2jk26_207418e2-9809-45a0-85a1-9baaf64716b8/router/0.log"
May 28 00:41:39.899868 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:39.899795    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4sppr_da2bacc2-9142-49d3-a23c-a62ba15e855e/serve-healthcheck-canary/0.log"
May 28 00:41:40.435199 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:40.435170    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ftvgh_e4375a32-e2b1-4f59-8e8e-1cfa96facde4/kube-rbac-proxy/0.log"
May 28 00:41:40.457943 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:40.457919    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ftvgh_e4375a32-e2b1-4f59-8e8e-1cfa96facde4/exporter/0.log"
May 28 00:41:40.480999 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:40.480977    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ftvgh_e4375a32-e2b1-4f59-8e8e-1cfa96facde4/extractor/0.log"
May 28 00:41:44.402319 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:44.402290    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-prqzq"
May 28 00:41:45.483859 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:45.483825    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-59tbq_1a8254b7-7a5c-4f77-9bf1-64605de1f147/kube-multus/0.log"
May 28 00:41:45.850992 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:45.850966    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tk77g_0a052ebe-16e4-40a8-b822-8160f7735c0b/kube-multus-additional-cni-plugins/0.log"
May 28 00:41:45.874437 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:45.874412    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tk77g_0a052ebe-16e4-40a8-b822-8160f7735c0b/egress-router-binary-copy/0.log"
May 28 00:41:45.899257 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:45.899228    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tk77g_0a052ebe-16e4-40a8-b822-8160f7735c0b/cni-plugins/0.log"
May 28 00:41:45.923113 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:45.923085    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tk77g_0a052ebe-16e4-40a8-b822-8160f7735c0b/bond-cni-plugin/0.log"
May 28 00:41:45.945622 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:45.945580    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tk77g_0a052ebe-16e4-40a8-b822-8160f7735c0b/routeoverride-cni/0.log"
May 28 00:41:45.969826 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:45.969793    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tk77g_0a052ebe-16e4-40a8-b822-8160f7735c0b/whereabouts-cni-bincopy/0.log"
May 28 00:41:45.994259 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:45.994234    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tk77g_0a052ebe-16e4-40a8-b822-8160f7735c0b/whereabouts-cni/0.log"
May 28 00:41:46.055818 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:46.055783    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2dnl7_c0d3ce5a-3b25-4284-8946-bbb3e5820ce6/network-metrics-daemon/0.log"
May 28 00:41:46.079356 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:46.079330    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2dnl7_c0d3ce5a-3b25-4284-8946-bbb3e5820ce6/kube-rbac-proxy/0.log"
May 28 00:41:47.672876 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:47.672848    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4zhg_3c9057d1-fd80-4b1c-af89-acfff9aec573/ovn-controller/0.log"
May 28 00:41:47.707299 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:47.707262    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4zhg_3c9057d1-fd80-4b1c-af89-acfff9aec573/ovn-acl-logging/0.log"
May 28 00:41:47.731639 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:47.731617    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4zhg_3c9057d1-fd80-4b1c-af89-acfff9aec573/kube-rbac-proxy-node/0.log"
May 28 00:41:47.754797 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:47.754771    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4zhg_3c9057d1-fd80-4b1c-af89-acfff9aec573/kube-rbac-proxy-ovn-metrics/0.log"
May 28 00:41:47.775331 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:47.775308    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4zhg_3c9057d1-fd80-4b1c-af89-acfff9aec573/northd/0.log"
May 28 00:41:47.800477 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:47.800455    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4zhg_3c9057d1-fd80-4b1c-af89-acfff9aec573/nbdb/0.log"
May 28 00:41:47.826907 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:47.826887    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4zhg_3c9057d1-fd80-4b1c-af89-acfff9aec573/sbdb/0.log"
May 28 00:41:48.005883 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:48.005808    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4zhg_3c9057d1-fd80-4b1c-af89-acfff9aec573/ovnkube-controller/0.log"
May 28 00:41:48.986264 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:48.986190    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-66fkn_bd4ec258-9332-443d-b262-f1b21d0e9337/check-endpoints/0.log"
May 28 00:41:49.038004 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:49.037971    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4sgrj_5dc8d16d-e5f7-410b-b433-0e30e23fd836/network-check-target-container/0.log"
May 28 00:41:49.873106 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:49.873078    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-crd6s_e6d96c6b-2765-4f69-934b-00641848c010/iptables-alerter/0.log"
May 28 00:41:50.498775 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:50.498748    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-f5hsm_777a4765-6b60-4801-95b7-f632242323c4/tuned/0.log"
May 28 00:41:53.062145 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:53.062113    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-m7mzw_abfdf3bc-d254-45a0-b65c-5a67c4490eb8/service-ca-operator/1.log"
May 28 00:41:53.064759 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:53.064733    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-m7mzw_abfdf3bc-d254-45a0-b65c-5a67c4490eb8/service-ca-operator/0.log"
May 28 00:41:53.754946 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:53.754862    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-d49tg_b4db5df0-40cc-4b4b-b339-ebf4c88e840a/csi-driver/0.log"
May 28 00:41:53.780452 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:53.780423    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-d49tg_b4db5df0-40cc-4b4b-b339-ebf4c88e840a/csi-node-driver-registrar/0.log"
May 28 00:41:53.813187 ip-10-0-142-230 kubenswrapper[2560]: I0528 00:41:53.813161    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-d49tg_b4db5df0-40cc-4b4b-b339-ebf4c88e840a/csi-liveness-probe/0.log"