Sep 09 06:30:50.672766 ip-10-0-128-45 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 06:30:50.672777 ip-10-0-128-45 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 06:30:50.672786 ip-10-0-128-45 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 06:30:50.673100 ip-10-0-128-45 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 06:31:00.729953 ip-10-0-128-45 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 09 06:31:00.729969 ip-10-0-128-45 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e4264b73d9b544ffb6f8190d882edc27 -- Sep 09 06:32:55.557257 ip-10-0-128-45 systemd[1]: Starting Kubernetes Kubelet... Sep 09 06:32:55.972873 ip-10-0-128-45 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 06:32:55.972873 ip-10-0-128-45 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 06:32:55.972873 ip-10-0-128-45 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 06:32:55.972873 ip-10-0-128-45 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 06:32:55.972873 ip-10-0-128-45 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 06:32:55.973505 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.972953 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 06:32:55.976572 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976557 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 06:32:55.976572 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976573 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976577 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976581 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976584 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976587 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976590 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976593 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976596 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976598 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976601 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976603 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976606 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976608 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976611 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976613 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976616 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976619 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976621 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976624 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976626 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 06:32:55.976634 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976629 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976636 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976639 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976642 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976645 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976648 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976650 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976653 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976657 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976661 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976664 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976666 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976669 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976672 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976674 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976676 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976679 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976681 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976684 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 06:32:55.977127 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976686 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976689 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976691 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976694 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976696 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976698 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976703 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976707 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976710 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976713 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976716 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976718 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976721 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976724 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976726 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976729 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976731 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976734 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976737 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976739 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 06:32:55.977590 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976742 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976744 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976747 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976749 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976752 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976754 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976757 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976759 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976761 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976764 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976766 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976769 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976771 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976774 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976776 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976779 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976781 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976784 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976786 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976790 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 06:32:55.978098 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976793 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976795 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976798 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976800 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976802 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976805 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976808 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976811 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976815 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976819 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976822 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976825 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976827 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976830 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976833 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976835 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976838 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976840 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.976843 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977533 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 06:32:55.978583 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977540 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977543 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977545 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977548 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977553 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977556 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977560 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977563 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977566 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977569 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977572 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977575 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977579 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977582 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977585 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977588 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977590 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977594 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977597 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 06:32:55.979124 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977600 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977603 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977606 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977608 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977611 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977613 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977616 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977618 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977621 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977624 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977626 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977629 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977631 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977634 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977637 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977639 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977642 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977644 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977647 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977649 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 06:32:55.979587 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977652 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977654 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977656 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977660 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977664 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977668 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977672 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977676 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977679 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977682 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977686 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977689 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977692 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977695 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977698 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977701 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977704 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977706 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977709 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 06:32:55.980088 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977711 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977714 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977716 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977719 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977721 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977724 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977726 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977729 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977731 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977733 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977736 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977739 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977741 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977743 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977746 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977748 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977751 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977754 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977757 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977759 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 06:32:55.980552 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977763 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977765 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977768 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977771 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977773 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977776 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977778 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977780 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977783 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977785 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977788 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977791 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977793 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977795 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977798 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977800 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977803 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977806 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977808 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.977810 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 06:32:55.981065 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978169 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978174 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978177 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978180 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978183 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978185 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978188 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978191 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978193 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978198 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978202 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978205 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978208 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978211 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978214 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978217 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978219 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978222 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978225 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 06:32:55.981557 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978227 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978230 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978233 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978236 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978239 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978241 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978244 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978247 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978249 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978251 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978254 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978256 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978259 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978261 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978264 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978266 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978269 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978272 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978274 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 06:32:55.982037 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978276 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978279 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978281 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978284 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978287 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978290 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978293 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978297 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978299 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978302 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978304 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978306 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978309 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978311 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978313 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978316 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978318 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978321 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978323 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978326 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 06:32:55.982595 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978328 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978331 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978335 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978338 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978341 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978344 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978346 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978348 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978351 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978353 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978356 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978358 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978361 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978363 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978366 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978370 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978373 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978375 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978378 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978380 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 06:32:55.983128 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978383 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978386 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978388 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978390 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978393 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978395 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978398 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978400 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978403 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978405 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978408 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978410 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978413 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978416 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978418 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978420 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978423 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978425 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978428 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978430 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 06:32:55.983614 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.978433 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978497 2563 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978530 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978535 2563 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978539 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978543 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978546 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978551 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978555 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978558 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978560 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978564 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978567 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978571 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978573 2563 flags.go:64] FLAG: --cgroup-root="" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978576 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978579 2563 flags.go:64] FLAG: --client-ca-file="" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978582 2563 flags.go:64] FLAG: --cloud-provider="external" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978585 2563 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978589 2563 flags.go:64] FLAG: --cluster-domain="" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978592 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978595 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978598 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978601 2563 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 06:32:55.984139 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978605 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978608 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978610 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978613 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978616 2563 flags.go:64] FLAG: --contention-profiling="false" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978619 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978622 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978625 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978628 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978631 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978637 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978640 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978643 2563 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978646 2563 flags.go:64] FLAG: --enable-server="true" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978648 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978653 2563 flags.go:64] FLAG: --event-burst="100" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978656 2563 flags.go:64] FLAG: --event-qps="50" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978658 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978661 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978664 2563 flags.go:64] FLAG: --eviction-hard="" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978668 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978670 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978674 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978677 2563 flags.go:64] FLAG: --eviction-soft="" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978679 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 06:32:55.984879 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978682 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978685 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978687 2563 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978690 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978693 2563 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978696 2563 flags.go:64] FLAG: --feature-gates="" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978708 2563 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978712 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978715 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978718 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978721 2563 flags.go:64] FLAG: --healthz-port="10248" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978724 2563 flags.go:64] FLAG: --help="false" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978727 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-128-45.ec2.internal" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978730 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978733 2563 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978735 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978739 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978743 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978746 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978749 2563 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978752 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978755 2563 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978758 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978761 2563 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 06:32:55.985952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978763 2563 flags.go:64] FLAG: --kube-reserved="" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978766 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978769 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978772 2563 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978775 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978777 2563 flags.go:64] FLAG: --lock-file="" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978781 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978784 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978787 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978792 2563 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978795 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978798 2563 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978800 2563 flags.go:64] FLAG: --logging-format="text" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978803 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978806 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978809 2563 flags.go:64] FLAG: --manifest-url="" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978811 2563 flags.go:64] FLAG: --manifest-url-header="" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978816 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978819 2563 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978822 2563 flags.go:64] FLAG: --max-pods="110" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978825 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978828 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978831 2563 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978834 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978837 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 06:32:55.987041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978840 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978843 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978850 2563 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978853 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978856 2563 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978858 2563 flags.go:64] FLAG: --pod-cidr="" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978861 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978867 2563 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978869 2563 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978872 2563 flags.go:64] FLAG: --pods-per-core="0" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978875 2563 flags.go:64] FLAG: --port="10250" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978878 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978880 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01397aadc4e7657ee" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978883 2563 flags.go:64] FLAG: --qos-reserved="" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978887 2563 flags.go:64] FLAG: --read-only-port="10255" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978890 2563 flags.go:64] FLAG: --register-node="true" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978892 2563 flags.go:64] FLAG: --register-with-taints="" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978896 2563 flags.go:64] FLAG: --registry-burst="10" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978899 2563 flags.go:64] FLAG: --registry-qps="5" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978902 2563 flags.go:64] FLAG: --reserved-cpus="" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978904 2563 flags.go:64] FLAG: --reserved-memory="" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978908 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978911 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978913 2563 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978916 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 06:32:55.988205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978919 2563 flags.go:64] FLAG: --runonce="false" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978922 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978925 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978929 2563 flags.go:64] FLAG: --seccomp-default="false" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978932 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978935 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978938 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978941 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978945 2563 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978948 2563 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978950 2563 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978953 2563 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978956 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978977 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978981 2563 flags.go:64] FLAG: --system-cgroups="" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978985 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978991 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978993 2563 flags.go:64] FLAG: --tls-cert-file="" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.978996 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979000 2563 flags.go:64] FLAG: --tls-min-version="" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979003 2563 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979006 2563 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979009 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979012 2563 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979015 2563 flags.go:64] FLAG: --v="2" Sep 09 06:32:55.988946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979018 2563 flags.go:64] FLAG: --version="false" Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979022 2563 flags.go:64] FLAG: --vmodule="" Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979025 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979028 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979133 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979136 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979140 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979142 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979145 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979148 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979153 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979156 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979159 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979161 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979164 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979167 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979170 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979172 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979175 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979178 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 06:32:55.989756 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979180 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979182 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979185 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979187 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979189 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979192 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979195 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979197 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979200 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979202 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979207 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979210 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979213 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979215 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979218 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979220 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979223 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979225 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979227 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 06:32:55.990613 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979229 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979232 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979234 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979238 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979240 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979243 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979245 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979248 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979251 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979254 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979256 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979259 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979261 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979263 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979266 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979269 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979271 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979274 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979276 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979278 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 06:32:55.991293 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979281 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979284 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979286 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979289 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979291 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979294 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979296 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979300 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979304 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979307 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979309 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979312 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979314 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979316 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979319 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979323 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979325 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979327 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979330 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 06:32:55.991812 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979333 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979336 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979340 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979343 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979346 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979349 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979352 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979354 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979357 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979360 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979363 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979365 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979368 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979371 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979374 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979376 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979379 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979381 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979384 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979388 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 06:32:55.992345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979391 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979393 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979396 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979398 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.979401 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.979406 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.986745 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.986763 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.986785 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.986793 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986871 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986878 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986884 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986889 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986893 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 06:32:55.992821 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986897 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986902 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986906 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986911 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986915 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986919 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986923 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986927 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986932 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986936 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986940 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986944 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986948 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986953 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986975 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986980 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986984 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986988 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986993 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 06:32:55.993278 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.986997 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987001 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987005 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987009 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987013 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987019 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987023 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987027 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987031 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987035 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987039 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987043 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987047 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987050 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987054 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987058 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987062 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987066 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987070 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987074 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 06:32:55.993773 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987078 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987082 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987088 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987093 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987097 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987101 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987105 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987109 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987113 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987117 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987124 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987130 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987134 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987139 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987144 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987148 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987153 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987158 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987163 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987167 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 06:32:55.994299 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987171 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987176 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987180 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987184 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987188 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987194 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987199 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987204 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987209 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987213 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987217 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987221 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987226 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987229 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987233 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987237 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987241 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987245 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987249 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987253 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 06:32:55.995076 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987257 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987261 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987265 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987269 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987273 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987277 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987281 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987285 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987289 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987293 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987297 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987302 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987306 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987310 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 06:32:55.995760 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987314 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.987322 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987511 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987519 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987523 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987527 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987531 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987535 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987539 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987543 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987547 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987551 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987556 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987560 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987564 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 06:32:55.996130 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987568 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987572 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987576 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987580 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987584 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987588 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987592 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987596 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987600 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987604 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987608 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987612 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987618 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987624 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987628 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987633 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987637 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987641 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987645 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987649 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 06:32:55.996594 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987654 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987658 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987662 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987665 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987669 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987674 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987677 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987681 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987685 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987689 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987693 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987697 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987701 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987705 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987709 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987713 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987716 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987720 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987725 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987729 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 06:32:55.997403 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987735 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987740 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987745 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987750 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987755 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987759 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987763 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987767 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987771 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987775 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987779 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987783 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987787 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987791 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987794 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987798 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987803 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987806 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987810 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987815 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 06:32:55.997885 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987820 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987823 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987827 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987831 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987835 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987839 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987843 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987847 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987851 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987855 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987859 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987863 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987867 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987871 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987875 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987882 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987888 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987892 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987897 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 06:32:55.998778 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987901 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987905 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987910 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987914 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987918 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987922 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:55.987926 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.987933 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.988079 2563 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.991148 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.992192 2563 server.go:1019] "Starting client certificate rotation" Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.992299 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.992353 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 06:32:55.999334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.995040 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 06:32:55.999712 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:55.998511 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 06:32:56.007770 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.007739 2563 log.go:25] "Validated CRI v1 runtime API" Sep 09 06:32:56.010823 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.010807 2563 log.go:25] "Validated CRI v1 image API" Sep 09 06:32:56.011750 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.011724 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 06:32:56.015553 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.015535 2563 fs.go:135] Filesystem UUIDs: map[20fa3343-7408-4c59-9498-c9ee685ff1f2:/dev/nvme0n1p4 71f8d84c-dad7-4a64-b042-285b4dbf5485:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 09 06:32:56.015602 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.015554 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 06:32:56.022321 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.022217 2563 manager.go:216] Machine: {Timestamp:2026-09-09 06:32:56.020537713 +0000 UTC m=+0.345348160 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090140 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec230dde7f5732e3de2da2d800245f00 SystemUUID:ec230dde-7f57-32e3-de2d-a2d800245f00 BootID:e4264b73-d9b5-44ff-b6f8-190d882edc27 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:88:ae:13:21:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:88:ae:13:21:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:b6:31:35:2b:6e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 06:32:56.022321 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.022320 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 06:32:56.022430 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.022395 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 06:32:56.023496 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.023475 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 06:32:56.024191 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.024171 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 06:32:56.024332 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.024196 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 06:32:56.024942 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.024933 2563 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 06:32:56.024994 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.024945 2563 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 06:32:56.024994 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.024954 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 06:32:56.024994 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.024979 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 06:32:56.026270 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.026260 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 06:32:56.029061 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.029050 2563 state_mem.go:36] "Initialized new in-memory state store" Sep 09 06:32:56.029172 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.029163 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 06:32:56.029209 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.029197 2563 kubelet.go:486] "Attempting to sync node with API server" Sep 09 06:32:56.029243 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.029214 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 06:32:56.029243 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.029228 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 06:32:56.029243 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.029238 2563 kubelet.go:398] "Adding apiserver pod source" Sep 09 06:32:56.029360 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.029252 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 06:32:56.029556 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.029545 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 06:32:56.029621 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.029563 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 06:32:56.031619 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.031602 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 06:32:56.033818 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.033804 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 06:32:56.033881 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.033834 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 06:32:56.034037 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034027 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 06:32:56.034074 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034043 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 06:32:56.034074 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034048 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 06:32:56.034074 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034054 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 06:32:56.034074 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034062 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 06:32:56.034074 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034068 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 06:32:56.034566 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034552 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 06:32:56.034627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034569 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 06:32:56.034627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034582 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 06:32:56.034627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034591 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 06:32:56.034627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034604 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 06:32:56.034627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.034617 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 06:32:56.035529 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.035517 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 06:32:56.035596 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.035532 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 06:32:56.039393 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.039375 2563 server.go:1268] "Started kubelet" Sep 09 06:32:56.039468 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.039415 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 06:32:56.039554 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.039500 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 06:32:56.039607 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.039574 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 06:32:56.040243 ip-10-0-128-45 systemd[1]: Started Kubernetes Kubelet. Sep 09 06:32:56.040900 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.040471 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 06:32:56.040900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.040578 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 06:32:56.040900 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.040598 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 06:32:56.041381 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.041164 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 06:32:56.041823 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.041804 2563 server.go:310] "Adding debug handlers to kubelet server" Sep 09 06:32:56.045105 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.045083 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 06:32:56.045547 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.045538 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 06:32:56.045606 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.045587 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 06:32:56.045686 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.045648 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.046132 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.046112 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 06:32:56.046236 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.046221 2563 factory.go:55] Registering systemd factory Sep 09 06:32:56.046292 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.046267 2563 factory.go:223] Registration of the systemd container factory successfully Sep 09 06:32:56.046552 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.046541 2563 factory.go:153] Registering CRI-O factory Sep 09 06:32:56.046592 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.046554 2563 factory.go:223] Registration of the crio container factory successfully Sep 09 06:32:56.046620 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.046593 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 06:32:56.046620 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.046606 2563 factory.go:103] Registering Raw factory Sep 09 06:32:56.046620 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.046615 2563 manager.go:1196] Started watching for new ooms in manager Sep 09 06:32:56.046890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.046880 2563 manager.go:349] Starting recovery of all containers Sep 09 06:32:56.047630 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.047613 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 06:32:56.047697 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.047634 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 06:32:56.047757 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.047721 2563 reconstruct.go:97] "Volume reconstruction finished" Sep 09 06:32:56.047757 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.047732 2563 reconciler.go:29] "Reconciler: start to sync state" Sep 09 06:32:56.047862 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.047828 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 06:32:56.048940 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.047853 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-45.ec2.internal.18d3936d9fe14422 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-45.ec2.internal,UID:ip-10-0-128-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-45.ec2.internal,},FirstTimestamp:2026-09-09 06:32:56.039277602 +0000 UTC m=+0.364088048,LastTimestamp:2026-09-09 06:32:56.039277602 +0000 UTC m=+0.364088048,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-45.ec2.internal,}" Sep 09 06:32:56.054530 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.054324 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 06:32:56.055019 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.055003 2563 manager.go:354] Recovery completed Sep 09 06:32:56.061889 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.061868 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 06:32:56.063627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.063606 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientMemory" Sep 09 06:32:56.063719 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.063644 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 06:32:56.063719 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.063661 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientPID" Sep 09 06:32:56.064311 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.064293 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 06:32:56.064311 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.064311 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 06:32:56.064440 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.064328 2563 state_mem.go:36] "Initialized new in-memory state store" Sep 09 06:32:56.064440 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.064327 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 06:32:56.065105 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.065091 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientMemory" Sep 09 06:32:56.065185 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.065121 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 06:32:56.065185 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.065136 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientPID" Sep 09 06:32:56.065580 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.065566 2563 policy_none.go:49] "None policy: Start" Sep 09 06:32:56.065634 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.065585 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 06:32:56.065634 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.065598 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 06:32:56.066531 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.066519 2563 policy_none.go:47] "Start" Sep 09 06:32:56.073345 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.073204 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vbzpk" Sep 09 06:32:56.075170 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.075105 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-45.ec2.internal.18d3936da154cddf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-45.ec2.internal,UID:ip-10-0-128-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-45.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-45.ec2.internal,},FirstTimestamp:2026-09-09 06:32:56.063626719 +0000 UTC m=+0.388437166,LastTimestamp:2026-09-09 06:32:56.063626719 +0000 UTC m=+0.388437166,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-45.ec2.internal,}" Sep 09 06:32:56.080539 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.080520 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vbzpk" Sep 09 06:32:56.101522 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.101506 2563 manager.go:337] "Starting Device Plugin manager" Sep 09 06:32:56.123095 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.101546 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 06:32:56.123095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.101559 2563 server.go:87] "Starting device plugin registration server" Sep 09 06:32:56.123095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.101837 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 06:32:56.123095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.101847 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 06:32:56.123095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.101947 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 06:32:56.123095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.102046 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 06:32:56.123095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.102053 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 06:32:56.123095 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.102551 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 06:32:56.123095 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.102602 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.140122 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.140094 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 06:32:56.141424 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.141411 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 06:32:56.141505 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.141461 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 06:32:56.141505 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.141484 2563 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 06:32:56.141603 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.141567 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 06:32:56.144312 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.144279 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 06:32:56.202658 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.202638 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 06:32:56.203566 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.203552 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientMemory" Sep 09 06:32:56.203638 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.203581 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 06:32:56.203638 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.203592 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientPID" Sep 09 06:32:56.203638 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.203613 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.213288 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.213269 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.213363 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.213290 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-45.ec2.internal\": node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.231753 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.231687 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.241775 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.241758 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal"] Sep 09 06:32:56.241835 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.241826 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 06:32:56.242564 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.242552 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientMemory" Sep 09 06:32:56.242614 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.242577 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 06:32:56.242614 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.242588 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientPID" Sep 09 06:32:56.244847 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.244835 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 06:32:56.245017 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.245004 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.245068 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.245038 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 06:32:56.245489 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.245474 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientMemory" Sep 09 06:32:56.245572 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.245502 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 06:32:56.245572 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.245519 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientPID" Sep 09 06:32:56.245572 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.245502 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientMemory" Sep 09 06:32:56.245677 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.245597 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 06:32:56.245677 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.245610 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientPID" Sep 09 06:32:56.247611 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.247598 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.247716 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.247621 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 06:32:56.248289 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.248274 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientMemory" Sep 09 06:32:56.248344 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.248303 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 06:32:56.248344 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.248314 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientPID" Sep 09 06:32:56.248569 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.248554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a9522f5749bff0fd7b7f335dc25acc3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal\" (UID: \"7a9522f5749bff0fd7b7f335dc25acc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.248608 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.248585 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a9522f5749bff0fd7b7f335dc25acc3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal\" (UID: \"7a9522f5749bff0fd7b7f335dc25acc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.248663 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.248643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce4b5beeab7b2f9c4597d30a91deced8-config\") pod \"kube-apiserver-proxy-ip-10-0-128-45.ec2.internal\" (UID: \"ce4b5beeab7b2f9c4597d30a91deced8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.269493 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.269477 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-45.ec2.internal\" not found" node="ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.273757 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.273742 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-45.ec2.internal\" not found" node="ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.332663 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.332648 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.349026 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.349008 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a9522f5749bff0fd7b7f335dc25acc3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal\" (UID: \"7a9522f5749bff0fd7b7f335dc25acc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.349073 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.349033 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a9522f5749bff0fd7b7f335dc25acc3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal\" (UID: \"7a9522f5749bff0fd7b7f335dc25acc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.349073 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.349050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce4b5beeab7b2f9c4597d30a91deced8-config\") pod \"kube-apiserver-proxy-ip-10-0-128-45.ec2.internal\" (UID: \"ce4b5beeab7b2f9c4597d30a91deced8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.349137 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.349085 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce4b5beeab7b2f9c4597d30a91deced8-config\") pod \"kube-apiserver-proxy-ip-10-0-128-45.ec2.internal\" (UID: \"ce4b5beeab7b2f9c4597d30a91deced8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.349137 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.349117 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a9522f5749bff0fd7b7f335dc25acc3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal\" (UID: \"7a9522f5749bff0fd7b7f335dc25acc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.349137 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.349128 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a9522f5749bff0fd7b7f335dc25acc3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal\" (UID: \"7a9522f5749bff0fd7b7f335dc25acc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.433178 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.433155 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.533918 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.533871 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.572400 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.572382 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.577844 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.577163 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal" Sep 09 06:32:56.634938 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.634911 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.735477 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.735446 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.835978 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.835895 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.936452 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:56.936429 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:56.991865 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.991841 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 06:32:56.992337 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:56.992001 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 06:32:57.037310 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:57.037289 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:57.082170 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.082149 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 06:32:57.083124 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.083096 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 06:27:56 +0000 UTC" deadline="2028-05-05 00:23:23.063132068 +0000 UTC" Sep 09 06:32:57.083124 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.083124 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14489h50m25.980011015s" Sep 09 06:32:57.101164 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.101150 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 06:32:57.102044 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.102022 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientMemory" Sep 09 06:32:57.102106 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.102057 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 06:32:57.102106 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.102067 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeHasSufficientPID" Sep 09 06:32:57.137724 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:57.137706 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-45.ec2.internal\" not found" Sep 09 06:32:57.189327 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.189308 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 06:32:57.197095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.197080 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 06:32:57.246616 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.246595 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal" Sep 09 06:32:57.256929 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.256910 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 06:32:57.257887 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.257876 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" Sep 09 06:32:57.271442 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.271427 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 06:32:57.354933 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:57.354882 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4b5beeab7b2f9c4597d30a91deced8.slice/crio-6748626f0c1d95b49752e1570468fa26d0cc5bb8f4c5df055dd4a33635ba6a0d WatchSource:0}: Error finding container 6748626f0c1d95b49752e1570468fa26d0cc5bb8f4c5df055dd4a33635ba6a0d: Status 404 returned error can't find the container with id 6748626f0c1d95b49752e1570468fa26d0cc5bb8f4c5df055dd4a33635ba6a0d Sep 09 06:32:57.356359 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:57.356245 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9522f5749bff0fd7b7f335dc25acc3.slice/crio-2fbe3f0452abe2a0a0f8111a054bc43ab0bfa4db9b49dd445e25a229655521bd WatchSource:0}: Error finding container 2fbe3f0452abe2a0a0f8111a054bc43ab0bfa4db9b49dd445e25a229655521bd: Status 404 returned error can't find the container with id 2fbe3f0452abe2a0a0f8111a054bc43ab0bfa4db9b49dd445e25a229655521bd Sep 09 06:32:57.362708 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:57.362682 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 06:32:58.031383 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.031031 2563 apiserver.go:52] "Watching apiserver" Sep 09 06:32:58.039563 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.039540 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 06:32:58.042337 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.042245 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-f952v","kube-system/konnectivity-agent-cq5mp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp","openshift-image-registry/node-ca-plfqc","openshift-multus/multus-additional-cni-plugins-cgq4b","openshift-multus/multus-th78r","openshift-multus/network-metrics-daemon-p8hkq","openshift-network-diagnostics/network-check-target-mhgvf","kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal","openshift-cluster-node-tuning-operator/tuned-fxtp9","openshift-dns/node-resolver-sv5qm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal","openshift-network-operator/iptables-alerter-gpwd6","openshift-ovn-kubernetes/ovnkube-node-vr8f9"] Sep 09 06:32:58.046349 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.045860 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:32:58.046349 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.045963 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:32:58.046349 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.046274 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 06:32:58.048671 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.048427 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.051376 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.051359 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.052213 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.052189 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 06:32:58.052331 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.052240 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 06:32:58.052467 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.052450 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 06:32:58.052588 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.052567 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5266r\"" Sep 09 06:32:58.053402 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.053382 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.053845 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.053823 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 06:32:58.053947 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.053902 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dnpnt\"" Sep 09 06:32:58.054220 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.054201 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 06:32:58.054404 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.054385 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 06:32:58.055902 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.055884 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.056270 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.056242 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 06:32:58.056358 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.056333 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 06:32:58.056622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.056606 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 06:32:58.056719 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.056664 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xzdv6\"" Sep 09 06:32:58.057700 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.057597 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbk7q\" (UniqueName: \"kubernetes.io/projected/cd36bba0-b482-4d51-9ab7-8b045c8f6006-kube-api-access-nbk7q\") pod \"iptables-alerter-gpwd6\" (UID: \"cd36bba0-b482-4d51-9ab7-8b045c8f6006\") " pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.057700 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.057638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:32:58.057700 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.057666 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpk2f\" (UniqueName: \"kubernetes.io/projected/00c134a2-09bb-4d7a-abaa-8a81a37d2459-kube-api-access-kpk2f\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:32:58.057923 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.057720 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd36bba0-b482-4d51-9ab7-8b045c8f6006-iptables-alerter-script\") pod \"iptables-alerter-gpwd6\" (UID: \"cd36bba0-b482-4d51-9ab7-8b045c8f6006\") " pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.057923 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.057756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd36bba0-b482-4d51-9ab7-8b045c8f6006-host-slash\") pod \"iptables-alerter-gpwd6\" (UID: \"cd36bba0-b482-4d51-9ab7-8b045c8f6006\") " pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.058343 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.058324 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-th78r" Sep 09 06:32:58.058814 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.058795 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vfc6c\"" Sep 09 06:32:58.058885 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.058838 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 06:32:58.058885 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.058869 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 06:32:58.059001 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.058896 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 06:32:58.060543 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.060520 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 06:32:58.060762 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.060745 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 06:32:58.060820 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.060779 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 06:32:58.061072 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.061049 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jhjzb\"" Sep 09 06:32:58.061393 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.061379 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.063692 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.063672 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.063816 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.063736 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:32:58.064203 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.064180 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 06:32:58.064411 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.064391 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 06:32:58.064490 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.064409 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 06:32:58.064679 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.064662 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 06:32:58.064755 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.064738 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 06:32:58.064917 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.064900 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6zrkl\"" Sep 09 06:32:58.064968 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.064945 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 06:32:58.065085 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.064665 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 06:32:58.068168 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.068144 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:32:58.070797 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.070768 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sv6jq\"" Sep 09 06:32:58.071069 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.071030 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 06:32:58.071545 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.071510 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 06:32:58.073237 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.073219 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.074387 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.074367 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.075645 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.075624 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 06:32:58.076269 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.076247 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wtgzr\"" Sep 09 06:32:58.076914 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.076878 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 06:32:58.077010 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.076922 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 06:32:58.077010 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.076989 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 06:32:58.077327 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.077308 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l9gkr\"" Sep 09 06:32:58.077587 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.077570 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:32:58.077670 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.077649 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:32:58.079420 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.079399 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q56vq" Sep 09 06:32:58.087349 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.087318 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q56vq" Sep 09 06:32:58.147775 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.147755 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 06:32:58.148705 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.148635 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" event={"ID":"7a9522f5749bff0fd7b7f335dc25acc3","Type":"ContainerStarted","Data":"2fbe3f0452abe2a0a0f8111a054bc43ab0bfa4db9b49dd445e25a229655521bd"} Sep 09 06:32:58.150144 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.150119 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal" event={"ID":"ce4b5beeab7b2f9c4597d30a91deced8","Type":"ContainerStarted","Data":"6748626f0c1d95b49752e1570468fa26d0cc5bb8f4c5df055dd4a33635ba6a0d"} Sep 09 06:32:58.158113 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158092 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-run-systemd\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.158212 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-run-openvswitch\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.158212 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158147 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-var-lib-cni-bin\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.158212 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158173 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj244\" (UniqueName: \"kubernetes.io/projected/4c5149eb-dfaa-4789-832f-e28de301bed9-kube-api-access-hj244\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.158212 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpk2f\" (UniqueName: \"kubernetes.io/projected/00c134a2-09bb-4d7a-abaa-8a81a37d2459-kube-api-access-kpk2f\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:32:58.158428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158228 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.158428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-systemd-units\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.158428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158277 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.158428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158303 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-sysctl-d\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.158428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158326 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-cnibin\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.158428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158349 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-run-k8s-cni-cncf-io\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.158428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158373 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:32:58.158428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158397 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb076d84-0bb2-474a-a6d7-c83e1c98904c-host\") pod \"node-ca-plfqc\" (UID: \"eb076d84-0bb2-474a-a6d7-c83e1c98904c\") " pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.158428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-var-lib-openvswitch\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158497 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-modprobe-d\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158520 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-sys\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158544 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-run-netns\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158568 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-daemon-config\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158592 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd36bba0-b482-4d51-9ab7-8b045c8f6006-host-slash\") pod \"iptables-alerter-gpwd6\" (UID: \"cd36bba0-b482-4d51-9ab7-8b045c8f6006\") " pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158620 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbk7q\" (UniqueName: \"kubernetes.io/projected/cd36bba0-b482-4d51-9ab7-8b045c8f6006-kube-api-access-nbk7q\") pod \"iptables-alerter-gpwd6\" (UID: \"cd36bba0-b482-4d51-9ab7-8b045c8f6006\") " pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-cni-bin\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158727 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-kubernetes\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158757 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-socket-dir-parent\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158782 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-slash\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158817 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-systemd\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.158861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158841 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgxk\" (UniqueName: \"kubernetes.io/projected/dd7ad407-6b70-4e58-aeb1-538a531a2067-kube-api-access-xqgxk\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158866 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-etc-kubernetes\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-registration-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158914 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-run-ovn\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158939 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-cni-dir\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158963 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-etc-selinux\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.158992 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-etc-openvswitch\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159014 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-dbus\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159058 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-sysconfig\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fac2dd48-8f12-4fe4-ae27-980d0d67772a-cni-binary-copy\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159110 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-var-lib-kubelet\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159136 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd36bba0-b482-4d51-9ab7-8b045c8f6006-iptables-alerter-script\") pod \"iptables-alerter-gpwd6\" (UID: \"cd36bba0-b482-4d51-9ab7-8b045c8f6006\") " pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-host\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159188 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd7ad407-6b70-4e58-aeb1-538a531a2067-tmp\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159213 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-run-multus-certs\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159242 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec-tmp-dir\") pod \"node-resolver-sv5qm\" (UID: \"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec\") " pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159268 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-run-netns\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.159600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159293 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-log-socket\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159320 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-ovn-node-metrics-cert\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159344 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-run\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159367 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-hostroot\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159420 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-cni-netd\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159446 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-conf-dir\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159471 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rh7\" (UniqueName: \"kubernetes.io/projected/3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec-kube-api-access-g4rh7\") pod \"node-resolver-sv5qm\" (UID: \"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec\") " pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159501 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-var-lib-kubelet\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159525 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-tuned\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrjn\" (UniqueName: \"kubernetes.io/projected/eb076d84-0bb2-474a-a6d7-c83e1c98904c-kube-api-access-ldrjn\") pod \"node-ca-plfqc\" (UID: \"eb076d84-0bb2-474a-a6d7-c83e1c98904c\") " pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159577 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-run-ovn-kubernetes\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159604 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ljf\" (UniqueName: \"kubernetes.io/projected/fac2dd48-8f12-4fe4-ae27-980d0d67772a-kube-api-access-w4ljf\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-device-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159742 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb076d84-0bb2-474a-a6d7-c83e1c98904c-serviceca\") pod \"node-ca-plfqc\" (UID: \"eb076d84-0bb2-474a-a6d7-c83e1c98904c\") " pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159791 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-node-log\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159819 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-ovnkube-config\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.160506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-env-overrides\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-sysctl-conf\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159958 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/084a6df0-80e1-466e-a084-78b251adc62f-agent-certs\") pod \"konnectivity-agent-cq5mp\" (UID: \"084a6df0-80e1-466e-a084-78b251adc62f\") " pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.159999 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ckk\" (UniqueName: \"kubernetes.io/projected/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-kube-api-access-f4ckk\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160026 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-kubelet\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160069 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc48t\" (UniqueName: \"kubernetes.io/projected/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-kube-api-access-jc48t\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160146 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-ovnkube-script-lib\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160173 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-kubelet-config\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-lib-modules\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160250 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-system-cni-dir\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160278 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/084a6df0-80e1-466e-a084-78b251adc62f-konnectivity-ca\") pod \"konnectivity-agent-cq5mp\" (UID: \"084a6df0-80e1-466e-a084-78b251adc62f\") " pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160324 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec-hosts-file\") pod \"node-resolver-sv5qm\" (UID: \"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec\") " pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160351 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-cnibin\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160394 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160421 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-var-lib-cni-multus\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.161423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160459 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-sys-fs\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.162284 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160485 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-system-cni-dir\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.162284 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.160511 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-socket-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.162284 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.161171 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd36bba0-b482-4d51-9ab7-8b045c8f6006-host-slash\") pod \"iptables-alerter-gpwd6\" (UID: \"cd36bba0-b482-4d51-9ab7-8b045c8f6006\") " pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.163239 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.162900 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:32:58.163239 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.162981 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs podName:00c134a2-09bb-4d7a-abaa-8a81a37d2459 nodeName:}" failed. No retries permitted until 2026-09-09 06:32:58.662945521 +0000 UTC m=+2.987755979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs") pod "network-metrics-daemon-p8hkq" (UID: "00c134a2-09bb-4d7a-abaa-8a81a37d2459") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:32:58.163441 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.163419 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd36bba0-b482-4d51-9ab7-8b045c8f6006-iptables-alerter-script\") pod \"iptables-alerter-gpwd6\" (UID: \"cd36bba0-b482-4d51-9ab7-8b045c8f6006\") " pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.167814 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.167423 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 06:32:58.172600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.172573 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpk2f\" (UniqueName: \"kubernetes.io/projected/00c134a2-09bb-4d7a-abaa-8a81a37d2459-kube-api-access-kpk2f\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:32:58.172990 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.172973 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbk7q\" (UniqueName: \"kubernetes.io/projected/cd36bba0-b482-4d51-9ab7-8b045c8f6006-kube-api-access-nbk7q\") pod \"iptables-alerter-gpwd6\" (UID: \"cd36bba0-b482-4d51-9ab7-8b045c8f6006\") " pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.261555 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261525 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.261555 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261557 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-modprobe-d\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.261720 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261573 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-sys\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.261720 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261593 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-run-netns\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.261720 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261617 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-daemon-config\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.261720 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261641 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.261720 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261664 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-cni-bin\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.261720 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261685 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-kubernetes\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.261720 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261695 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-sys\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.261720 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-socket-dir-parent\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.261720 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.261704 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261737 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-modprobe-d\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261772 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-run-netns\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261752 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-slash\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261819 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-kubernetes\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.261861 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret podName:e9bc0891-f9b1-469c-9337-c7832bb1d7ae nodeName:}" failed. No retries permitted until 2026-09-09 06:32:58.761818566 +0000 UTC m=+3.086629007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret") pod "global-pull-secret-syncer-f952v" (UID: "e9bc0891-f9b1-469c-9337-c7832bb1d7ae") : object "kube-system"/"original-pull-secret" not registered Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-systemd\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgxk\" (UniqueName: \"kubernetes.io/projected/dd7ad407-6b70-4e58-aeb1-538a531a2067-kube-api-access-xqgxk\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261945 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-socket-dir-parent\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.261951 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-etc-kubernetes\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-registration-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262039 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-run-ovn\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-cni-dir\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-etc-selinux\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262161 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-cni-bin\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262206 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-systemd\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262236 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-cni-dir\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262251 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-etc-selinux\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262291 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-registration-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-etc-openvswitch\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262340 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-dbus\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.262366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262357 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-sysconfig\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262397 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fac2dd48-8f12-4fe4-ae27-980d0d67772a-cni-binary-copy\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262433 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-etc-openvswitch\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262450 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-var-lib-kubelet\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262472 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-dbus\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262485 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-host\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262403 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-slash\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262536 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-host\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262541 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-daemon-config\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262525 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-run-ovn\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262569 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-var-lib-kubelet\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262573 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-etc-kubernetes\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd7ad407-6b70-4e58-aeb1-538a531a2067-tmp\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-run-multus-certs\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262647 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-sysconfig\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec-tmp-dir\") pod \"node-resolver-sv5qm\" (UID: \"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec\") " pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262694 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-run-multus-certs\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.262798 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262704 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-run-netns\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262730 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-log-socket\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262757 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-ovn-node-metrics-cert\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262788 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-run-netns\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262778 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-run\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262840 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-run\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262855 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-hostroot\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262897 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-log-socket\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-cni-netd\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262946 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fac2dd48-8f12-4fe4-ae27-980d0d67772a-cni-binary-copy\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262954 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-cni-netd\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-conf-dir\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.262978 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-multus-conf-dir\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263007 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec-tmp-dir\") pod \"node-resolver-sv5qm\" (UID: \"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec\") " pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263023 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-hostroot\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rh7\" (UniqueName: \"kubernetes.io/projected/3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec-kube-api-access-g4rh7\") pod \"node-resolver-sv5qm\" (UID: \"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec\") " pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263079 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.263636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263067 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-var-lib-kubelet\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263120 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-tuned\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263150 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrjn\" (UniqueName: \"kubernetes.io/projected/eb076d84-0bb2-474a-a6d7-c83e1c98904c-kube-api-access-ldrjn\") pod \"node-ca-plfqc\" (UID: \"eb076d84-0bb2-474a-a6d7-c83e1c98904c\") " pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263155 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-var-lib-kubelet\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-run-ovn-kubernetes\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ljf\" (UniqueName: \"kubernetes.io/projected/fac2dd48-8f12-4fe4-ae27-980d0d67772a-kube-api-access-w4ljf\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263238 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-device-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb076d84-0bb2-474a-a6d7-c83e1c98904c-serviceca\") pod \"node-ca-plfqc\" (UID: \"eb076d84-0bb2-474a-a6d7-c83e1c98904c\") " pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-node-log\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-ovnkube-config\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-env-overrides\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263375 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-sysctl-conf\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263403 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/084a6df0-80e1-466e-a084-78b251adc62f-agent-certs\") pod \"konnectivity-agent-cq5mp\" (UID: \"084a6df0-80e1-466e-a084-78b251adc62f\") " pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ckk\" (UniqueName: \"kubernetes.io/projected/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-kube-api-access-f4ckk\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263457 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-kubelet\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc48t\" (UniqueName: \"kubernetes.io/projected/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-kube-api-access-jc48t\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263521 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-ovnkube-script-lib\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.264727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263546 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-kubelet-config\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-lib-modules\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-system-cni-dir\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263604 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-run-ovn-kubernetes\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/084a6df0-80e1-466e-a084-78b251adc62f-konnectivity-ca\") pod \"konnectivity-agent-cq5mp\" (UID: \"084a6df0-80e1-466e-a084-78b251adc62f\") " pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263642 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-node-log\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263647 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec-hosts-file\") pod \"node-resolver-sv5qm\" (UID: \"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec\") " pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263687 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-device-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263696 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-sysctl-conf\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263707 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec-hosts-file\") pod \"node-resolver-sv5qm\" (UID: \"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec\") " pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263710 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-cnibin\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.263729 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-cnibin\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264141 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb076d84-0bb2-474a-a6d7-c83e1c98904c-serviceca\") pod \"node-ca-plfqc\" (UID: \"eb076d84-0bb2-474a-a6d7-c83e1c98904c\") " pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264295 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-kubelet\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264314 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-ovnkube-config\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264359 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264388 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-var-lib-cni-multus\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264414 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-sys-fs\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.265622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264426 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-env-overrides\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264438 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-system-cni-dir\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264486 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-system-cni-dir\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264512 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-lib-modules\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264577 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-system-cni-dir\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264616 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-socket-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264627 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264644 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-run-systemd\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-run-openvswitch\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264675 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-var-lib-cni-multus\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264708 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-var-lib-cni-bin\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264740 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj244\" (UniqueName: \"kubernetes.io/projected/4c5149eb-dfaa-4789-832f-e28de301bed9-kube-api-access-hj244\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264800 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-systemd-units\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264828 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264857 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-sysctl-d\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264882 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-cnibin\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.266662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264909 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-ovnkube-script-lib\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264909 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-run-k8s-cni-cncf-io\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264939 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-run-k8s-cni-cncf-io\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264984 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb076d84-0bb2-474a-a6d7-c83e1c98904c-host\") pod \"node-ca-plfqc\" (UID: \"eb076d84-0bb2-474a-a6d7-c83e1c98904c\") " pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.264992 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-sys-fs\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265127 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c5149eb-dfaa-4789-832f-e28de301bed9-socket-dir\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265188 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-run-openvswitch\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265239 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb076d84-0bb2-474a-a6d7-c83e1c98904c-host\") pod \"node-ca-plfqc\" (UID: \"eb076d84-0bb2-474a-a6d7-c83e1c98904c\") " pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265241 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-run-systemd\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-var-lib-openvswitch\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265266 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/084a6df0-80e1-466e-a084-78b251adc62f-konnectivity-ca\") pod \"konnectivity-agent-cq5mp\" (UID: \"084a6df0-80e1-466e-a084-78b251adc62f\") " pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265311 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-var-lib-openvswitch\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265330 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-systemd-units\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265357 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-sysctl-d\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265418 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-kubelet-config\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.267523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265406 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.268307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265487 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-cnibin\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.268307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265521 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fac2dd48-8f12-4fe4-ae27-980d0d67772a-host-var-lib-cni-bin\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.268307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265650 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.268307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.265726 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.268307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.267285 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/084a6df0-80e1-466e-a084-78b251adc62f-agent-certs\") pod \"konnectivity-agent-cq5mp\" (UID: \"084a6df0-80e1-466e-a084-78b251adc62f\") " pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:32:58.268307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.267430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd7ad407-6b70-4e58-aeb1-538a531a2067-tmp\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.268307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.267984 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-ovn-node-metrics-cert\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.268307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.268192 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd7ad407-6b70-4e58-aeb1-538a531a2067-etc-tuned\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.275571 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.274777 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 06:32:58.275571 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.274806 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 06:32:58.275571 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.274820 2563 projected.go:196] Error preparing data for projected volume kube-api-access-672fg for pod openshift-network-diagnostics/network-check-target-mhgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:32:58.275571 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.274879 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg podName:2b9d7dfe-79ff-4abf-a090-7df31ea02020 nodeName:}" failed. No retries permitted until 2026-09-09 06:32:58.774861343 +0000 UTC m=+3.099671788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-672fg" (UniqueName: "kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg") pod "network-check-target-mhgvf" (UID: "2b9d7dfe-79ff-4abf-a090-7df31ea02020") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:32:58.275831 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.275644 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgxk\" (UniqueName: \"kubernetes.io/projected/dd7ad407-6b70-4e58-aeb1-538a531a2067-kube-api-access-xqgxk\") pod \"tuned-fxtp9\" (UID: \"dd7ad407-6b70-4e58-aeb1-538a531a2067\") " pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.276863 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.276835 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ckk\" (UniqueName: \"kubernetes.io/projected/c588c0a2-8028-45ef-bf1a-01f4c8a167b1-kube-api-access-f4ckk\") pod \"multus-additional-cni-plugins-cgq4b\" (UID: \"c588c0a2-8028-45ef-bf1a-01f4c8a167b1\") " pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.277935 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.277898 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc48t\" (UniqueName: \"kubernetes.io/projected/a249fd1c-7ac3-43b4-b802-4e8cb39b1a89-kube-api-access-jc48t\") pod \"ovnkube-node-vr8f9\" (UID: \"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89\") " pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.278656 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.278628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ljf\" (UniqueName: \"kubernetes.io/projected/fac2dd48-8f12-4fe4-ae27-980d0d67772a-kube-api-access-w4ljf\") pod \"multus-th78r\" (UID: \"fac2dd48-8f12-4fe4-ae27-980d0d67772a\") " pod="openshift-multus/multus-th78r" Sep 09 06:32:58.278987 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.278950 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrjn\" (UniqueName: \"kubernetes.io/projected/eb076d84-0bb2-474a-a6d7-c83e1c98904c-kube-api-access-ldrjn\") pod \"node-ca-plfqc\" (UID: \"eb076d84-0bb2-474a-a6d7-c83e1c98904c\") " pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.279071 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.279002 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj244\" (UniqueName: \"kubernetes.io/projected/4c5149eb-dfaa-4789-832f-e28de301bed9-kube-api-access-hj244\") pod \"aws-ebs-csi-driver-node-zv7dp\" (UID: \"4c5149eb-dfaa-4789-832f-e28de301bed9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.280489 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.280467 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rh7\" (UniqueName: \"kubernetes.io/projected/3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec-kube-api-access-g4rh7\") pod \"node-resolver-sv5qm\" (UID: \"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec\") " pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.367522 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.367495 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gpwd6" Sep 09 06:32:58.378233 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.378210 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" Sep 09 06:32:58.379288 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:58.379230 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd36bba0_b482_4d51_9ab7_8b045c8f6006.slice/crio-c9b4daea39c8b17ac2848877d177dd6fe4b6cf9978b555d472a3033192c2306b WatchSource:0}: Error finding container c9b4daea39c8b17ac2848877d177dd6fe4b6cf9978b555d472a3033192c2306b: Status 404 returned error can't find the container with id c9b4daea39c8b17ac2848877d177dd6fe4b6cf9978b555d472a3033192c2306b Sep 09 06:32:58.387933 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:58.387899 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc588c0a2_8028_45ef_bf1a_01f4c8a167b1.slice/crio-8d8fa322fbffb2710d18decfdb3d30667383fa935f7c79b31680c9b100920986 WatchSource:0}: Error finding container 8d8fa322fbffb2710d18decfdb3d30667383fa935f7c79b31680c9b100920986: Status 404 returned error can't find the container with id 8d8fa322fbffb2710d18decfdb3d30667383fa935f7c79b31680c9b100920986 Sep 09 06:32:58.388885 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.388865 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" Sep 09 06:32:58.399873 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:58.399851 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5149eb_dfaa_4789_832f_e28de301bed9.slice/crio-bcf8193c9111eb32d602704f464357cbc6e6777ece183f1c2d4281eb10e44737 WatchSource:0}: Error finding container bcf8193c9111eb32d602704f464357cbc6e6777ece183f1c2d4281eb10e44737: Status 404 returned error can't find the container with id bcf8193c9111eb32d602704f464357cbc6e6777ece183f1c2d4281eb10e44737 Sep 09 06:32:58.408114 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.408092 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-plfqc" Sep 09 06:32:58.410314 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.410288 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-th78r" Sep 09 06:32:58.417873 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:58.417851 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb076d84_0bb2_474a_a6d7_c83e1c98904c.slice/crio-c9b540c61fe70c4c45f937994788eb95fa45c80477bda3efe72b49b7a9a05fe1 WatchSource:0}: Error finding container c9b540c61fe70c4c45f937994788eb95fa45c80477bda3efe72b49b7a9a05fe1: Status 404 returned error can't find the container with id c9b540c61fe70c4c45f937994788eb95fa45c80477bda3efe72b49b7a9a05fe1 Sep 09 06:32:58.420135 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.420003 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:32:58.420671 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:58.420624 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac2dd48_8f12_4fe4_ae27_980d0d67772a.slice/crio-abed7195fdfa15ec766dcf5152e65c58c50624826d858466be13098d574954c0 WatchSource:0}: Error finding container abed7195fdfa15ec766dcf5152e65c58c50624826d858466be13098d574954c0: Status 404 returned error can't find the container with id abed7195fdfa15ec766dcf5152e65c58c50624826d858466be13098d574954c0 Sep 09 06:32:58.431067 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.430249 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:32:58.437600 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.437571 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" Sep 09 06:32:58.443634 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.443611 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sv5qm" Sep 09 06:32:58.455251 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:32:58.455228 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3527edfe_7c0e_4ab9_8357_d7e4e1fee8ec.slice/crio-aca8c7b39d83a497c308e6629d63ca3d047670f138ce26095b68a2f799ff4776 WatchSource:0}: Error finding container aca8c7b39d83a497c308e6629d63ca3d047670f138ce26095b68a2f799ff4776: Status 404 returned error can't find the container with id aca8c7b39d83a497c308e6629d63ca3d047670f138ce26095b68a2f799ff4776 Sep 09 06:32:58.538572 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.538520 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 06:32:58.669011 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.668919 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:32:58.669151 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.669102 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:32:58.669217 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.669179 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs podName:00c134a2-09bb-4d7a-abaa-8a81a37d2459 nodeName:}" failed. No retries permitted until 2026-09-09 06:32:59.669156674 +0000 UTC m=+3.993967119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs") pod "network-metrics-daemon-p8hkq" (UID: "00c134a2-09bb-4d7a-abaa-8a81a37d2459") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:32:58.770141 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.770106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:58.770293 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.770237 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 06:32:58.770354 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.770307 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret podName:e9bc0891-f9b1-469c-9337-c7832bb1d7ae nodeName:}" failed. No retries permitted until 2026-09-09 06:32:59.770287794 +0000 UTC m=+4.095098254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret") pod "global-pull-secret-syncer-f952v" (UID: "e9bc0891-f9b1-469c-9337-c7832bb1d7ae") : object "kube-system"/"original-pull-secret" not registered Sep 09 06:32:58.870906 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:58.870878 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:32:58.871085 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.871066 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 06:32:58.871085 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.871083 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 06:32:58.871167 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.871092 2563 projected.go:196] Error preparing data for projected volume kube-api-access-672fg for pod openshift-network-diagnostics/network-check-target-mhgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:32:58.871167 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:58.871142 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg podName:2b9d7dfe-79ff-4abf-a090-7df31ea02020 nodeName:}" failed. No retries permitted until 2026-09-09 06:32:59.871125241 +0000 UTC m=+4.195935694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-672fg" (UniqueName: "kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg") pod "network-check-target-mhgvf" (UID: "2b9d7dfe-79ff-4abf-a090-7df31ea02020") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:32:59.090587 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.090496 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 06:27:58 +0000 UTC" deadline="2028-04-25 22:36:08.101131492 +0000 UTC" Sep 09 06:32:59.090587 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.090536 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14272h3m9.010598038s" Sep 09 06:32:59.153871 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.153835 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sv5qm" event={"ID":"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec","Type":"ContainerStarted","Data":"aca8c7b39d83a497c308e6629d63ca3d047670f138ce26095b68a2f799ff4776"} Sep 09 06:32:59.155122 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.155096 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" event={"ID":"dd7ad407-6b70-4e58-aeb1-538a531a2067","Type":"ContainerStarted","Data":"704ec4205515f5356158e2ad39d7d9c774d748571e059348e7da399122ccb89c"} Sep 09 06:32:59.156851 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.156823 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq5mp" event={"ID":"084a6df0-80e1-466e-a084-78b251adc62f","Type":"ContainerStarted","Data":"474c61ba99be907afd8ca81e800047abf17d6f2b3607e8f281eb31c1d7df8680"} Sep 09 06:32:59.158234 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.158208 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" event={"ID":"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89","Type":"ContainerStarted","Data":"1009f7292555014822f6892d078e22475590a3c21ac4ec8b8a6dcb7fc31f572d"} Sep 09 06:32:59.160239 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.160214 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gpwd6" event={"ID":"cd36bba0-b482-4d51-9ab7-8b045c8f6006","Type":"ContainerStarted","Data":"c9b4daea39c8b17ac2848877d177dd6fe4b6cf9978b555d472a3033192c2306b"} Sep 09 06:32:59.161509 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.161485 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-th78r" event={"ID":"fac2dd48-8f12-4fe4-ae27-980d0d67772a","Type":"ContainerStarted","Data":"abed7195fdfa15ec766dcf5152e65c58c50624826d858466be13098d574954c0"} Sep 09 06:32:59.162921 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.162898 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-plfqc" event={"ID":"eb076d84-0bb2-474a-a6d7-c83e1c98904c","Type":"ContainerStarted","Data":"c9b540c61fe70c4c45f937994788eb95fa45c80477bda3efe72b49b7a9a05fe1"} Sep 09 06:32:59.164312 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.164291 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" event={"ID":"4c5149eb-dfaa-4789-832f-e28de301bed9","Type":"ContainerStarted","Data":"bcf8193c9111eb32d602704f464357cbc6e6777ece183f1c2d4281eb10e44737"} Sep 09 06:32:59.165628 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.165602 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" event={"ID":"c588c0a2-8028-45ef-bf1a-01f4c8a167b1","Type":"ContainerStarted","Data":"8d8fa322fbffb2710d18decfdb3d30667383fa935f7c79b31680c9b100920986"} Sep 09 06:32:59.677681 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.677024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:32:59.677681 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:59.677228 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:32:59.677681 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:59.677291 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs podName:00c134a2-09bb-4d7a-abaa-8a81a37d2459 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:01.677272971 +0000 UTC m=+6.002083408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs") pod "network-metrics-daemon-p8hkq" (UID: "00c134a2-09bb-4d7a-abaa-8a81a37d2459") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:32:59.777576 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.777549 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:32:59.777727 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:59.777706 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 06:32:59.777810 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:59.777768 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret podName:e9bc0891-f9b1-469c-9337-c7832bb1d7ae nodeName:}" failed. No retries permitted until 2026-09-09 06:33:01.777751909 +0000 UTC m=+6.102562343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret") pod "global-pull-secret-syncer-f952v" (UID: "e9bc0891-f9b1-469c-9337-c7832bb1d7ae") : object "kube-system"/"original-pull-secret" not registered Sep 09 06:32:59.879070 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:32:59.878866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:32:59.879172 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:59.879095 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 06:32:59.879172 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:59.879117 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 06:32:59.879172 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:59.879128 2563 projected.go:196] Error preparing data for projected volume kube-api-access-672fg for pod openshift-network-diagnostics/network-check-target-mhgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:32:59.879269 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:32:59.879174 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg podName:2b9d7dfe-79ff-4abf-a090-7df31ea02020 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:01.879160897 +0000 UTC m=+6.203971331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-672fg" (UniqueName: "kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg") pod "network-check-target-mhgvf" (UID: "2b9d7dfe-79ff-4abf-a090-7df31ea02020") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:33:00.091758 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:00.091711 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 06:27:58 +0000 UTC" deadline="2028-03-23 20:29:48.77699873 +0000 UTC" Sep 09 06:33:00.091758 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:00.091754 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13477h56m48.68524914s" Sep 09 06:33:00.145059 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:00.142081 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:00.145059 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:00.142201 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:00.145059 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:00.142594 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:00.145059 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:00.142687 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:00.145059 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:00.144900 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:00.145059 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:00.145015 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:00.172726 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:00.172696 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal" event={"ID":"ce4b5beeab7b2f9c4597d30a91deced8","Type":"ContainerStarted","Data":"15ce4750669ea5b3d39c5a0845ab725e37c9b4c6dc64e28b73e39742093d1033"} Sep 09 06:33:01.203082 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:01.202628 2563 generic.go:356] "Generic (PLEG): container finished" podID="7a9522f5749bff0fd7b7f335dc25acc3" containerID="55cfded37aee1ddfcac1f7ff1793ad1fc657c1afc184f1795ba4cc6f0dc42c98" exitCode=0 Sep 09 06:33:01.203082 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:01.202771 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" event={"ID":"7a9522f5749bff0fd7b7f335dc25acc3","Type":"ContainerDied","Data":"55cfded37aee1ddfcac1f7ff1793ad1fc657c1afc184f1795ba4cc6f0dc42c98"} Sep 09 06:33:01.222122 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:01.222025 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-45.ec2.internal" podStartSLOduration=4.222005295 podStartE2EDuration="4.222005295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:00.187839233 +0000 UTC m=+4.512649691" watchObservedRunningTime="2026-09-09 06:33:01.222005295 +0000 UTC m=+5.546815751" Sep 09 06:33:01.695476 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:01.695285 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:01.695476 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:01.695448 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:33:01.695722 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:01.695517 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs podName:00c134a2-09bb-4d7a-abaa-8a81a37d2459 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:05.695497638 +0000 UTC m=+10.020308077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs") pod "network-metrics-daemon-p8hkq" (UID: "00c134a2-09bb-4d7a-abaa-8a81a37d2459") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:33:01.796603 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:01.796554 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:01.796814 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:01.796790 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 06:33:01.796890 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:01.796878 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret podName:e9bc0891-f9b1-469c-9337-c7832bb1d7ae nodeName:}" failed. No retries permitted until 2026-09-09 06:33:05.796857796 +0000 UTC m=+10.121668245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret") pod "global-pull-secret-syncer-f952v" (UID: "e9bc0891-f9b1-469c-9337-c7832bb1d7ae") : object "kube-system"/"original-pull-secret" not registered Sep 09 06:33:01.897322 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:01.897283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:01.897504 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:01.897484 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 06:33:01.897574 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:01.897510 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 06:33:01.897574 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:01.897525 2563 projected.go:196] Error preparing data for projected volume kube-api-access-672fg for pod openshift-network-diagnostics/network-check-target-mhgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:33:01.897668 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:01.897582 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg podName:2b9d7dfe-79ff-4abf-a090-7df31ea02020 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:05.897565207 +0000 UTC m=+10.222375648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-672fg" (UniqueName: "kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg") pod "network-check-target-mhgvf" (UID: "2b9d7dfe-79ff-4abf-a090-7df31ea02020") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:33:02.144217 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:02.144182 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:02.144367 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:02.144313 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:02.144719 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:02.144690 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:02.144804 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:02.144785 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:02.144843 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:02.144837 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:02.144976 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:02.144901 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:04.143119 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:04.143057 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:04.143119 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:04.143072 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:04.143670 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:04.143204 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:04.143670 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:04.143330 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:04.143670 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:04.143400 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:04.143670 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:04.143476 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:05.731669 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:05.731637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:05.732124 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:05.731822 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:33:05.732124 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:05.731888 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs podName:00c134a2-09bb-4d7a-abaa-8a81a37d2459 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:13.731870834 +0000 UTC m=+18.056681292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs") pod "network-metrics-daemon-p8hkq" (UID: "00c134a2-09bb-4d7a-abaa-8a81a37d2459") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:33:05.833003 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:05.832955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:05.833187 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:05.833113 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 06:33:05.833187 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:05.833172 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret podName:e9bc0891-f9b1-469c-9337-c7832bb1d7ae nodeName:}" failed. No retries permitted until 2026-09-09 06:33:13.833153581 +0000 UTC m=+18.157964020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret") pod "global-pull-secret-syncer-f952v" (UID: "e9bc0891-f9b1-469c-9337-c7832bb1d7ae") : object "kube-system"/"original-pull-secret" not registered Sep 09 06:33:05.933503 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:05.933467 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:05.933677 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:05.933654 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 06:33:05.933677 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:05.933672 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 06:33:05.933772 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:05.933684 2563 projected.go:196] Error preparing data for projected volume kube-api-access-672fg for pod openshift-network-diagnostics/network-check-target-mhgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:33:05.933772 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:05.933741 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg podName:2b9d7dfe-79ff-4abf-a090-7df31ea02020 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:13.933722347 +0000 UTC m=+18.258532785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-672fg" (UniqueName: "kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg") pod "network-check-target-mhgvf" (UID: "2b9d7dfe-79ff-4abf-a090-7df31ea02020") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:33:06.143022 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:06.142986 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:06.143185 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:06.143027 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:06.143185 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:06.143123 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:06.143302 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:06.143203 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:06.143302 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:06.143253 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:06.143431 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:06.143395 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:08.142707 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:08.142679 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:08.143225 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:08.142733 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:08.143225 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:08.142742 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:08.143225 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:08.142816 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:08.143225 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:08.142883 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:08.143225 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:08.142982 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:10.144735 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:10.144705 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:10.145104 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:10.144755 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:10.145104 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:10.144806 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:10.145104 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:10.144820 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:10.145104 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:10.144880 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:10.145104 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:10.144947 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:12.142127 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:12.142089 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:12.142523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:12.142093 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:12.142523 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:12.142221 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:12.142523 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:12.142265 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:12.142523 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:12.142345 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:12.142523 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:12.142420 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:13.792166 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:13.792133 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:13.792732 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:13.792298 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:33:13.792732 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:13.792377 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs podName:00c134a2-09bb-4d7a-abaa-8a81a37d2459 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:29.792355992 +0000 UTC m=+34.117166428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs") pod "network-metrics-daemon-p8hkq" (UID: "00c134a2-09bb-4d7a-abaa-8a81a37d2459") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:33:13.892692 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:13.892658 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:13.892834 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:13.892815 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 06:33:13.892906 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:13.892893 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret podName:e9bc0891-f9b1-469c-9337-c7832bb1d7ae nodeName:}" failed. No retries permitted until 2026-09-09 06:33:29.892871192 +0000 UTC m=+34.217681638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret") pod "global-pull-secret-syncer-f952v" (UID: "e9bc0891-f9b1-469c-9337-c7832bb1d7ae") : object "kube-system"/"original-pull-secret" not registered Sep 09 06:33:13.993901 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:13.993875 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:13.994067 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:13.994046 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 06:33:13.994138 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:13.994075 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 06:33:13.994138 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:13.994088 2563 projected.go:196] Error preparing data for projected volume kube-api-access-672fg for pod openshift-network-diagnostics/network-check-target-mhgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:33:13.994223 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:13.994146 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg podName:2b9d7dfe-79ff-4abf-a090-7df31ea02020 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:29.994128651 +0000 UTC m=+34.318939099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-672fg" (UniqueName: "kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg") pod "network-check-target-mhgvf" (UID: "2b9d7dfe-79ff-4abf-a090-7df31ea02020") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:33:14.145835 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:14.145424 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:14.145835 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:14.145458 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:14.145835 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:14.145522 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:14.145835 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:14.145541 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:14.145835 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:14.145591 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:14.145835 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:14.145700 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:16.142669 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:16.142637 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:16.143143 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:16.142727 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:16.143143 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:16.142731 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:16.143143 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:16.142859 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:16.143143 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:16.142868 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:16.143143 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:16.142947 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:18.142415 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.142251 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:18.142805 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.142251 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:18.142805 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:18.142491 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:18.142805 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:18.142591 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:18.142805 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.142259 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:18.142805 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:18.142699 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:18.238936 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.238914 2563 generic.go:356] "Generic (PLEG): container finished" podID="c588c0a2-8028-45ef-bf1a-01f4c8a167b1" containerID="b03d7237f362196f4859762f9abb3a41d647ef41bcbf7440443f96aed8bb5c54" exitCode=0 Sep 09 06:33:18.239072 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.238999 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" event={"ID":"c588c0a2-8028-45ef-bf1a-01f4c8a167b1","Type":"ContainerDied","Data":"b03d7237f362196f4859762f9abb3a41d647ef41bcbf7440443f96aed8bb5c54"} Sep 09 06:33:18.242869 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.242838 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" event={"ID":"7a9522f5749bff0fd7b7f335dc25acc3","Type":"ContainerStarted","Data":"1515e9d1f058e087dd9eefd678c9a62e83a0ab727fb2c88e60e06ec66227d3e4"} Sep 09 06:33:18.244507 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.244473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sv5qm" event={"ID":"3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec","Type":"ContainerStarted","Data":"79818a8f04aa951151282404fef6681f7d86cd0c0747a2fa3200eea87cc92a1a"} Sep 09 06:33:18.246187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.246167 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" event={"ID":"dd7ad407-6b70-4e58-aeb1-538a531a2067","Type":"ContainerStarted","Data":"e4af7fb78fe4f14d6a982704f50c0a7f0b89c0f8ff0c4d9b3d04068772d7ec37"} Sep 09 06:33:18.247664 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.247647 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq5mp" event={"ID":"084a6df0-80e1-466e-a084-78b251adc62f","Type":"ContainerStarted","Data":"c038414e2ade6301064ea1dbc88b11996caab49ebd948d6ecc78efa6a4c51c8b"} Sep 09 06:33:18.251088 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.251067 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" event={"ID":"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89","Type":"ContainerStarted","Data":"548bdeeca91b509f901334208b1f886cd2c869a01441a201cd4ae572c5eee1ce"} Sep 09 06:33:18.251184 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.251138 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" event={"ID":"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89","Type":"ContainerStarted","Data":"8bd1d00d2fcc0f56769c877559de2076e620a67b158d9ca10944536295c3fa7e"} Sep 09 06:33:18.251264 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.251187 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" event={"ID":"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89","Type":"ContainerStarted","Data":"64220ffbabb817aa5e1cff068f55d54c68bacb1719105b033ccb7ab0483f28d6"} Sep 09 06:33:18.251264 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.251239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" event={"ID":"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89","Type":"ContainerStarted","Data":"b39e362eaf2670f4bebaf6de5cdb473906b4e29cb85148022cda91ff2f8d80b8"} Sep 09 06:33:18.251346 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.251284 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" event={"ID":"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89","Type":"ContainerStarted","Data":"3ecdf2067b3be82ad71eb44d222959abf06f37146bd84a724a9e7bc3c41f6f06"} Sep 09 06:33:18.252675 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.252643 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-th78r" event={"ID":"fac2dd48-8f12-4fe4-ae27-980d0d67772a","Type":"ContainerStarted","Data":"dbafa3efe65f583abd5ab641b6ecd7f0bcd03272633e044c4d6f5fc63580d3e2"} Sep 09 06:33:18.254217 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.254180 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-plfqc" event={"ID":"eb076d84-0bb2-474a-a6d7-c83e1c98904c","Type":"ContainerStarted","Data":"427c9c465310e8b0828e12b0f3b0305b417ffa731db38894b42f813947f3ce8a"} Sep 09 06:33:18.255853 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.255836 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" event={"ID":"4c5149eb-dfaa-4789-832f-e28de301bed9","Type":"ContainerStarted","Data":"ff90703773a33b71c139c62485269c913acd1105f9ad47bd0aec76c1b2bd9e45"} Sep 09 06:33:18.275515 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.275480 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-th78r" podStartSLOduration=2.98209457 podStartE2EDuration="22.275467679s" totalImagesPullingTime="19.293373109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:18.275115211 +0000 UTC m=+22.599925666" watchObservedRunningTime="2026-09-09 06:33:18.275467679 +0000 UTC m=+22.600278135" Sep 09 06:33:18.291269 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.291236 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fxtp9" podStartSLOduration=3.136541698 podStartE2EDuration="22.291225095s" totalImagesPullingTime="19.154683397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:18.290765734 +0000 UTC m=+22.615576189" watchObservedRunningTime="2026-09-09 06:33:18.291225095 +0000 UTC m=+22.616035551" Sep 09 06:33:18.304196 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.304163 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-plfqc" podStartSLOduration=3.188609739 podStartE2EDuration="22.304155626s" totalImagesPullingTime="19.115545887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:18.303809884 +0000 UTC m=+22.628620340" watchObservedRunningTime="2026-09-09 06:33:18.304155626 +0000 UTC m=+22.628966082" Sep 09 06:33:18.324412 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.324381 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cq5mp" podStartSLOduration=3.263898122 podStartE2EDuration="22.324369974s" totalImagesPullingTime="19.060471852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:18.323878329 +0000 UTC m=+22.648688785" watchObservedRunningTime="2026-09-09 06:33:18.324369974 +0000 UTC m=+22.649180430" Sep 09 06:33:18.347817 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:18.347773 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sv5qm" podStartSLOduration=3.170834709 podStartE2EDuration="22.347761959s" totalImagesPullingTime="19.17692725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:18.347381005 +0000 UTC m=+22.672191461" watchObservedRunningTime="2026-09-09 06:33:18.347761959 +0000 UTC m=+22.672572415" Sep 09 06:33:19.261143 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:19.261108 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" event={"ID":"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89","Type":"ContainerStarted","Data":"9c4bfb5012054f19b371854722f7f0d1a2f9e4ce2ccc0e96cf0f80c2f780cda6"} Sep 09 06:33:19.262507 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:19.262474 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gpwd6" event={"ID":"cd36bba0-b482-4d51-9ab7-8b045c8f6006","Type":"ContainerStarted","Data":"611f847fcf2afe9a5bd1e6ab00b23eca2fac3b87d1662c1337dd75ed85369d4c"} Sep 09 06:33:19.276901 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:19.276865 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-45.ec2.internal" podStartSLOduration=22.276852601 podStartE2EDuration="22.276852601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:18.364075709 +0000 UTC m=+22.688886182" watchObservedRunningTime="2026-09-09 06:33:19.276852601 +0000 UTC m=+23.601663058" Sep 09 06:33:19.277013 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:19.276928 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gpwd6" podStartSLOduration=4.079879346 podStartE2EDuration="23.276923744s" totalImagesPullingTime="19.197044398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:19.276578531 +0000 UTC m=+23.601388987" watchObservedRunningTime="2026-09-09 06:33:19.276923744 +0000 UTC m=+23.601734199" Sep 09 06:33:19.313784 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:19.313745 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 06:33:20.115069 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:20.114934 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T06:33:19.313767767Z","UUID":"0743622c-3dfb-48ed-ab64-5ca059f61709","Handler":null,"Name":"","Endpoint":""} Sep 09 06:33:20.116794 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:20.116771 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 06:33:20.116903 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:20.116804 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 06:33:20.142900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:20.142622 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:20.142900 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:20.142745 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:20.142900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:20.142634 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:20.142900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:20.142786 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:20.142900 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:20.142867 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:20.143148 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:20.142942 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:20.270122 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:20.270089 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" event={"ID":"4c5149eb-dfaa-4789-832f-e28de301bed9","Type":"ContainerStarted","Data":"db3d14c5adf03af17d6779d3580b4e4364d7b66c69fcf58f4a37268f93c015ad"} Sep 09 06:33:21.274835 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:21.274795 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" event={"ID":"4c5149eb-dfaa-4789-832f-e28de301bed9","Type":"ContainerStarted","Data":"68efdba14f6ec1098364a3c3aa7421bbce1e25f6b6a4351c65f12ec112b8df6a"} Sep 09 06:33:21.278443 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:21.278419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" event={"ID":"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89","Type":"ContainerStarted","Data":"3f5184c30d2ec1ab1c66f102bcd2360f774e884dc9d9de47b2a07696e5e9db29"} Sep 09 06:33:21.292153 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:21.292112 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv7dp" podStartSLOduration=3.464510177 podStartE2EDuration="25.292098792s" totalImagesPullingTime="21.827588615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:21.291813797 +0000 UTC m=+25.616624446" watchObservedRunningTime="2026-09-09 06:33:21.292098792 +0000 UTC m=+25.616909247" Sep 09 06:33:22.142197 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:22.142029 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:22.142348 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:22.142031 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:22.142348 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:22.142290 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:22.142459 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:22.142101 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:22.142459 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:22.142386 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:22.142553 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:22.142495 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:22.905313 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:22.905279 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:33:22.905905 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:22.905881 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:33:23.283074 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:23.283006 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:33:23.283393 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:23.283376 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cq5mp" Sep 09 06:33:24.142152 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.142126 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:24.142541 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.142126 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:24.142541 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:24.142250 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:24.142541 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.142126 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:24.142541 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:24.142344 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:24.142541 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:24.142435 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:24.288275 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.287992 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" event={"ID":"a249fd1c-7ac3-43b4-b802-4e8cb39b1a89","Type":"ContainerStarted","Data":"fbd1ac723684648ebb393b4e4222f1911d7411357d8f63ed900ec5ba857be1ae"} Sep 09 06:33:24.288373 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.288322 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:33:24.288421 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.288380 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:33:24.288421 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.288408 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:33:24.290173 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.290149 2563 generic.go:356] "Generic (PLEG): container finished" podID="c588c0a2-8028-45ef-bf1a-01f4c8a167b1" containerID="96be6631177f7396f72cdcdb70cd023586533406ed6fc7066d1b508c24f5fa57" exitCode=0 Sep 09 06:33:24.290287 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.290230 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" event={"ID":"c588c0a2-8028-45ef-bf1a-01f4c8a167b1","Type":"ContainerDied","Data":"96be6631177f7396f72cdcdb70cd023586533406ed6fc7066d1b508c24f5fa57"} Sep 09 06:33:24.303941 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.303925 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:33:24.304048 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.304042 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:33:24.324200 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:24.324156 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" podStartSLOduration=9.05600452 podStartE2EDuration="28.32414345s" totalImagesPullingTime="19.26813893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:24.322927122 +0000 UTC m=+28.647737577" watchObservedRunningTime="2026-09-09 06:33:24.32414345 +0000 UTC m=+28.648953888" Sep 09 06:33:25.804299 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:25.804272 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f952v"] Sep 09 06:33:25.804896 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:25.804412 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:25.804896 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:25.804498 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:25.807199 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:25.807176 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mhgvf"] Sep 09 06:33:25.807306 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:25.807279 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:25.807389 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:25.807372 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:25.810432 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:25.810414 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8hkq"] Sep 09 06:33:25.810532 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:25.810521 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:25.810619 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:25.810598 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:26.297050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:26.297010 2563 generic.go:356] "Generic (PLEG): container finished" podID="c588c0a2-8028-45ef-bf1a-01f4c8a167b1" containerID="3e43111b06ff960767dfa486788e393f786725722cbe7735cbbca910ccb00481" exitCode=0 Sep 09 06:33:26.297200 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:26.297088 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" event={"ID":"c588c0a2-8028-45ef-bf1a-01f4c8a167b1","Type":"ContainerDied","Data":"3e43111b06ff960767dfa486788e393f786725722cbe7735cbbca910ccb00481"} Sep 09 06:33:27.142737 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:27.142561 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:27.143033 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:27.142585 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:27.143033 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:27.142829 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:27.143033 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:27.142895 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:28.141950 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:28.141921 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:28.142141 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:28.142038 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:28.303831 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:28.303800 2563 generic.go:356] "Generic (PLEG): container finished" podID="c588c0a2-8028-45ef-bf1a-01f4c8a167b1" containerID="56fabecbaea1e2ccb0e2d6ac953cb2554e077f56abbe48cdbd6a668f0cc78725" exitCode=0 Sep 09 06:33:28.304182 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:28.303852 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" event={"ID":"c588c0a2-8028-45ef-bf1a-01f4c8a167b1","Type":"ContainerDied","Data":"56fabecbaea1e2ccb0e2d6ac953cb2554e077f56abbe48cdbd6a668f0cc78725"} Sep 09 06:33:29.141683 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:29.141651 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:29.141837 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:29.141692 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:29.141837 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:29.141793 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:33:29.141924 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:29.141906 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f952v" podUID="e9bc0891-f9b1-469c-9337-c7832bb1d7ae" Sep 09 06:33:29.811150 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:29.811113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:29.811533 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:29.811256 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:33:29.811533 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:29.811321 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs podName:00c134a2-09bb-4d7a-abaa-8a81a37d2459 nodeName:}" failed. No retries permitted until 2026-09-09 06:34:01.811300342 +0000 UTC m=+66.136110788 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs") pod "network-metrics-daemon-p8hkq" (UID: "00c134a2-09bb-4d7a-abaa-8a81a37d2459") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 06:33:29.911660 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:29.911628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:29.911919 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:29.911894 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 06:33:29.911993 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:29.911984 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret podName:e9bc0891-f9b1-469c-9337-c7832bb1d7ae nodeName:}" failed. No retries permitted until 2026-09-09 06:34:01.911950384 +0000 UTC m=+66.236760821 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret") pod "global-pull-secret-syncer-f952v" (UID: "e9bc0891-f9b1-469c-9337-c7832bb1d7ae") : object "kube-system"/"original-pull-secret" not registered Sep 09 06:33:30.012629 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.012599 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:30.012801 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.012747 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 06:33:30.012801 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.012770 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 06:33:30.012801 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.012782 2563 projected.go:196] Error preparing data for projected volume kube-api-access-672fg for pod openshift-network-diagnostics/network-check-target-mhgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:33:30.013006 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.012839 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg podName:2b9d7dfe-79ff-4abf-a090-7df31ea02020 nodeName:}" failed. No retries permitted until 2026-09-09 06:34:02.012822309 +0000 UTC m=+66.337632749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-672fg" (UniqueName: "kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg") pod "network-check-target-mhgvf" (UID: "2b9d7dfe-79ff-4abf-a090-7df31ea02020") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 06:33:30.142675 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.142644 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:30.142837 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.142772 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhgvf" podUID="2b9d7dfe-79ff-4abf-a090-7df31ea02020" Sep 09 06:33:30.498406 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.498343 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-45.ec2.internal" event="NodeReady" Sep 09 06:33:30.498579 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.498453 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 06:33:30.532026 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.531997 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69496f8585-qwx6b"] Sep 09 06:33:30.548161 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.548077 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d8257"] Sep 09 06:33:30.548303 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.548265 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.550939 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.550916 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 06:33:30.551077 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.550917 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8bmbn\"" Sep 09 06:33:30.551150 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.551130 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 06:33:30.551205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.551158 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 06:33:30.557682 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.557660 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 06:33:30.560863 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.560842 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.561269 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.561254 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69496f8585-qwx6b"] Sep 09 06:33:30.561327 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.561299 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q8t8b"] Sep 09 06:33:30.563243 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.563224 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 06:33:30.563405 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.563383 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 06:33:30.563496 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.563442 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgrss\"" Sep 09 06:33:30.571654 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.571637 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d8257"] Sep 09 06:33:30.571734 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.571667 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8t8b"] Sep 09 06:33:30.571776 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.571738 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8t8b" Sep 09 06:33:30.574018 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.573999 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 06:33:30.574119 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.574100 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vmmtb\"" Sep 09 06:33:30.574182 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.574104 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 06:33:30.574440 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.574423 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 06:33:30.717753 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/784894d0-7e1e-4611-9e85-c737ddbe3cad-config-volume\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.717930 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717765 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvggj\" (UniqueName: \"kubernetes.io/projected/784894d0-7e1e-4611-9e85-c737ddbe3cad-kube-api-access-tvggj\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.717930 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-trusted-ca\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.717930 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717829 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784894d0-7e1e-4611-9e85-c737ddbe3cad-tmp-dir\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.717930 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc547\" (UniqueName: \"kubernetes.io/projected/02718a56-6748-4bb7-9cef-01d5fe02132a-kube-api-access-dc547\") pod \"ingress-canary-q8t8b\" (UID: \"02718a56-6748-4bb7-9cef-01d5fe02132a\") " pod="openshift-ingress-canary/ingress-canary-q8t8b" Sep 09 06:33:30.717930 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717902 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2500df92-212c-4be0-8813-972d17dc1952-ca-trust-extracted\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.717930 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-image-registry-private-configuration\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.718187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717936 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-bound-sa-token\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.718187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717956 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-installation-pull-secrets\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.718187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.717996 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52h9p\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-kube-api-access-52h9p\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.718187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.718020 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.718187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.718039 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.718187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.718082 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-registry-certificates\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.718187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.718098 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02718a56-6748-4bb7-9cef-01d5fe02132a-cert\") pod \"ingress-canary-q8t8b\" (UID: \"02718a56-6748-4bb7-9cef-01d5fe02132a\") " pod="openshift-ingress-canary/ingress-canary-q8t8b" Sep 09 06:33:30.819089 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819027 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/784894d0-7e1e-4611-9e85-c737ddbe3cad-config-volume\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.819089 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvggj\" (UniqueName: \"kubernetes.io/projected/784894d0-7e1e-4611-9e85-c737ddbe3cad-kube-api-access-tvggj\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.819089 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819086 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-trusted-ca\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.819584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819193 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784894d0-7e1e-4611-9e85-c737ddbe3cad-tmp-dir\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.819584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc547\" (UniqueName: \"kubernetes.io/projected/02718a56-6748-4bb7-9cef-01d5fe02132a-kube-api-access-dc547\") pod \"ingress-canary-q8t8b\" (UID: \"02718a56-6748-4bb7-9cef-01d5fe02132a\") " pod="openshift-ingress-canary/ingress-canary-q8t8b" Sep 09 06:33:30.819584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2500df92-212c-4be0-8813-972d17dc1952-ca-trust-extracted\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.819584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-image-registry-private-configuration\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.819584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819319 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-bound-sa-token\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.819584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819351 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-installation-pull-secrets\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.819584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819510 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784894d0-7e1e-4611-9e85-c737ddbe3cad-tmp-dir\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.819584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819562 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52h9p\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-kube-api-access-52h9p\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819655 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2500df92-212c-4be0-8813-972d17dc1952-ca-trust-extracted\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819658 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-registry-certificates\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.819705 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02718a56-6748-4bb7-9cef-01d5fe02132a-cert\") pod \"ingress-canary-q8t8b\" (UID: \"02718a56-6748-4bb7-9cef-01d5fe02132a\") " pod="openshift-ingress-canary/ingress-canary-q8t8b" Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.819717 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.819734 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69496f8585-qwx6b: secret "image-registry-tls" not found Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.819759 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls podName:784894d0-7e1e-4611-9e85-c737ddbe3cad nodeName:}" failed. No retries permitted until 2026-09-09 06:33:31.319742575 +0000 UTC m=+35.644553024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls") pod "dns-default-d8257" (UID: "784894d0-7e1e-4611-9e85-c737ddbe3cad") : secret "dns-default-metrics-tls" not found Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:30.819785 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls podName:2500df92-212c-4be0-8813-972d17dc1952 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:31.319768489 +0000 UTC m=+35.644578925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls") pod "image-registry-69496f8585-qwx6b" (UID: "2500df92-212c-4be0-8813-972d17dc1952") : secret "image-registry-tls" not found Sep 09 06:33:30.819995 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.819623 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/784894d0-7e1e-4611-9e85-c737ddbe3cad-config-volume\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.820435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.820094 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-trusted-ca\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.820435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.820167 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-registry-certificates\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.823855 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.823828 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-image-registry-private-configuration\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.823951 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.823856 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-installation-pull-secrets\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.823951 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.823837 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02718a56-6748-4bb7-9cef-01d5fe02132a-cert\") pod \"ingress-canary-q8t8b\" (UID: \"02718a56-6748-4bb7-9cef-01d5fe02132a\") " pod="openshift-ingress-canary/ingress-canary-q8t8b" Sep 09 06:33:30.828099 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.828075 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc547\" (UniqueName: \"kubernetes.io/projected/02718a56-6748-4bb7-9cef-01d5fe02132a-kube-api-access-dc547\") pod \"ingress-canary-q8t8b\" (UID: \"02718a56-6748-4bb7-9cef-01d5fe02132a\") " pod="openshift-ingress-canary/ingress-canary-q8t8b" Sep 09 06:33:30.828594 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.828475 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvggj\" (UniqueName: \"kubernetes.io/projected/784894d0-7e1e-4611-9e85-c737ddbe3cad-kube-api-access-tvggj\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:30.829031 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.829010 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52h9p\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-kube-api-access-52h9p\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.829134 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.829119 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-bound-sa-token\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:30.879128 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:30.878909 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8t8b" Sep 09 06:33:31.142168 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:31.142139 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:33:31.142383 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:31.142180 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:33:31.144930 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:31.144909 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 06:33:31.148568 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:31.145350 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 06:33:31.148568 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:31.145562 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gdzz9\"" Sep 09 06:33:31.190594 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:31.190568 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8t8b"] Sep 09 06:33:31.195904 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:33:31.195883 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02718a56_6748_4bb7_9cef_01d5fe02132a.slice/crio-6daab9b14364131f42c984d9ee0b18955ac7e3f2275661673b8fab941b900f03 WatchSource:0}: Error finding container 6daab9b14364131f42c984d9ee0b18955ac7e3f2275661673b8fab941b900f03: Status 404 returned error can't find the container with id 6daab9b14364131f42c984d9ee0b18955ac7e3f2275661673b8fab941b900f03 Sep 09 06:33:31.311814 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:31.311787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8t8b" event={"ID":"02718a56-6748-4bb7-9cef-01d5fe02132a","Type":"ContainerStarted","Data":"6daab9b14364131f42c984d9ee0b18955ac7e3f2275661673b8fab941b900f03"} Sep 09 06:33:31.325159 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:31.325131 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:31.325293 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:31.325173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:31.325293 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:31.325264 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 06:33:31.325293 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:31.325274 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69496f8585-qwx6b: secret "image-registry-tls" not found Sep 09 06:33:31.325440 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:31.325328 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls podName:2500df92-212c-4be0-8813-972d17dc1952 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:32.325315064 +0000 UTC m=+36.650125498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls") pod "image-registry-69496f8585-qwx6b" (UID: "2500df92-212c-4be0-8813-972d17dc1952") : secret "image-registry-tls" not found Sep 09 06:33:31.325440 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:31.325339 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 06:33:31.325440 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:31.325410 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls podName:784894d0-7e1e-4611-9e85-c737ddbe3cad nodeName:}" failed. No retries permitted until 2026-09-09 06:33:32.325393091 +0000 UTC m=+36.650203543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls") pod "dns-default-d8257" (UID: "784894d0-7e1e-4611-9e85-c737ddbe3cad") : secret "dns-default-metrics-tls" not found Sep 09 06:33:32.142439 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:32.142192 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:33:32.145279 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:32.145256 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 06:33:32.145279 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:32.145271 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6vz4x\"" Sep 09 06:33:32.145463 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:32.145256 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 06:33:32.330593 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:32.330565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:32.330593 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:32.330596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:32.330824 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:32.330708 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 06:33:32.330824 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:32.330718 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 06:33:32.330824 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:32.330722 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69496f8585-qwx6b: secret "image-registry-tls" not found Sep 09 06:33:32.330824 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:32.330789 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls podName:784894d0-7e1e-4611-9e85-c737ddbe3cad nodeName:}" failed. No retries permitted until 2026-09-09 06:33:34.330767539 +0000 UTC m=+38.655577977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls") pod "dns-default-d8257" (UID: "784894d0-7e1e-4611-9e85-c737ddbe3cad") : secret "dns-default-metrics-tls" not found Sep 09 06:33:32.330824 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:32.330809 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls podName:2500df92-212c-4be0-8813-972d17dc1952 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:34.330798319 +0000 UTC m=+38.655608756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls") pod "image-registry-69496f8585-qwx6b" (UID: "2500df92-212c-4be0-8813-972d17dc1952") : secret "image-registry-tls" not found Sep 09 06:33:33.320517 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:33.320477 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" event={"ID":"c588c0a2-8028-45ef-bf1a-01f4c8a167b1","Type":"ContainerStarted","Data":"8551e31184fd90a257312ad55b34da17cef6d2ddb441d8bf79d1cd8e34baff3a"} Sep 09 06:33:34.326169 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:34.326141 2563 generic.go:356] "Generic (PLEG): container finished" podID="c588c0a2-8028-45ef-bf1a-01f4c8a167b1" containerID="8551e31184fd90a257312ad55b34da17cef6d2ddb441d8bf79d1cd8e34baff3a" exitCode=0 Sep 09 06:33:34.326584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:34.326197 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" event={"ID":"c588c0a2-8028-45ef-bf1a-01f4c8a167b1","Type":"ContainerDied","Data":"8551e31184fd90a257312ad55b34da17cef6d2ddb441d8bf79d1cd8e34baff3a"} Sep 09 06:33:34.346431 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:34.346405 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:34.346535 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:34.346445 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:34.346595 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:34.346549 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 06:33:34.346595 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:34.346562 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 06:33:34.346595 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:34.346577 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69496f8585-qwx6b: secret "image-registry-tls" not found Sep 09 06:33:34.346735 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:34.346611 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls podName:784894d0-7e1e-4611-9e85-c737ddbe3cad nodeName:}" failed. No retries permitted until 2026-09-09 06:33:38.346589596 +0000 UTC m=+42.671400031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls") pod "dns-default-d8257" (UID: "784894d0-7e1e-4611-9e85-c737ddbe3cad") : secret "dns-default-metrics-tls" not found Sep 09 06:33:34.346735 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:34.346630 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls podName:2500df92-212c-4be0-8813-972d17dc1952 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:38.346621194 +0000 UTC m=+42.671431630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls") pod "image-registry-69496f8585-qwx6b" (UID: "2500df92-212c-4be0-8813-972d17dc1952") : secret "image-registry-tls" not found Sep 09 06:33:35.330806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:35.330773 2563 generic.go:356] "Generic (PLEG): container finished" podID="c588c0a2-8028-45ef-bf1a-01f4c8a167b1" containerID="65e7d826f3f62d72a0c30058495e604bd692120f0d4654d8e4deacf4794d288c" exitCode=0 Sep 09 06:33:35.331254 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:35.330822 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" event={"ID":"c588c0a2-8028-45ef-bf1a-01f4c8a167b1","Type":"ContainerDied","Data":"65e7d826f3f62d72a0c30058495e604bd692120f0d4654d8e4deacf4794d288c"} Sep 09 06:33:36.336131 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:36.336088 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" event={"ID":"c588c0a2-8028-45ef-bf1a-01f4c8a167b1","Type":"ContainerStarted","Data":"afe02f9aec03d59d76d582426e67b292fe5d9794575585731f615ebe326b3d25"} Sep 09 06:33:36.337445 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:36.337420 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8t8b" event={"ID":"02718a56-6748-4bb7-9cef-01d5fe02132a","Type":"ContainerStarted","Data":"88823509ebee4c44c4ca79a074c53cd496eafb2dd4b0fba4d94ec24ed2000b8d"} Sep 09 06:33:36.359638 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:36.359591 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cgq4b" podStartSLOduration=8.22546462 podStartE2EDuration="40.359578006s" totalImagesPullingTime="32.134113386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:36.358824436 +0000 UTC m=+40.683634895" watchObservedRunningTime="2026-09-09 06:33:36.359578006 +0000 UTC m=+40.684388462" Sep 09 06:33:36.373757 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:36.373717 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q8t8b" podStartSLOduration=1.906711977 podStartE2EDuration="6.373706259s" totalImagesPullingTime="4.466994282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:33:36.372887409 +0000 UTC m=+40.697697867" watchObservedRunningTime="2026-09-09 06:33:36.373706259 +0000 UTC m=+40.698516705" Sep 09 06:33:38.375076 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:38.375041 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:38.375076 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:38.375075 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:38.375470 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:38.375176 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 06:33:38.375470 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:38.375177 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 06:33:38.375470 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:38.375187 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69496f8585-qwx6b: secret "image-registry-tls" not found Sep 09 06:33:38.375470 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:38.375245 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls podName:784894d0-7e1e-4611-9e85-c737ddbe3cad nodeName:}" failed. No retries permitted until 2026-09-09 06:33:46.375229957 +0000 UTC m=+50.700040391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls") pod "dns-default-d8257" (UID: "784894d0-7e1e-4611-9e85-c737ddbe3cad") : secret "dns-default-metrics-tls" not found Sep 09 06:33:38.375470 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:38.375259 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls podName:2500df92-212c-4be0-8813-972d17dc1952 nodeName:}" failed. No retries permitted until 2026-09-09 06:33:46.375252956 +0000 UTC m=+50.700063391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls") pod "image-registry-69496f8585-qwx6b" (UID: "2500df92-212c-4be0-8813-972d17dc1952") : secret "image-registry-tls" not found Sep 09 06:33:46.429025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:46.428984 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:33:46.429025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:46.429026 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:33:46.429432 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:46.429122 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 06:33:46.429432 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:46.429126 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 06:33:46.429432 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:46.429141 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69496f8585-qwx6b: secret "image-registry-tls" not found Sep 09 06:33:46.429432 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:46.429188 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls podName:784894d0-7e1e-4611-9e85-c737ddbe3cad nodeName:}" failed. No retries permitted until 2026-09-09 06:34:02.429170322 +0000 UTC m=+66.753980768 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls") pod "dns-default-d8257" (UID: "784894d0-7e1e-4611-9e85-c737ddbe3cad") : secret "dns-default-metrics-tls" not found Sep 09 06:33:46.429432 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:33:46.429201 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls podName:2500df92-212c-4be0-8813-972d17dc1952 nodeName:}" failed. No retries permitted until 2026-09-09 06:34:02.429195687 +0000 UTC m=+66.754006122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls") pod "image-registry-69496f8585-qwx6b" (UID: "2500df92-212c-4be0-8813-972d17dc1952") : secret "image-registry-tls" not found Sep 09 06:33:56.318745 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:33:56.318709 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vr8f9" Sep 09 06:34:01.832532 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:01.832499 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:34:01.835608 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:01.835590 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 06:34:01.843073 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:01.843056 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 06:34:01.843131 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:01.843120 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs podName:00c134a2-09bb-4d7a-abaa-8a81a37d2459 nodeName:}" failed. No retries permitted until 2026-09-09 06:35:05.843100997 +0000 UTC m=+130.167911431 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs") pod "network-metrics-daemon-p8hkq" (UID: "00c134a2-09bb-4d7a-abaa-8a81a37d2459") : secret "metrics-daemon-secret" not found Sep 09 06:34:01.932787 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:01.932759 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:34:01.936358 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:01.936331 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 06:34:01.946666 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:01.946646 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9bc0891-f9b1-469c-9337-c7832bb1d7ae-original-pull-secret\") pod \"global-pull-secret-syncer-f952v\" (UID: \"e9bc0891-f9b1-469c-9337-c7832bb1d7ae\") " pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:34:02.033304 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.033281 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:34:02.035829 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.035813 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 06:34:02.046149 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.046132 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 06:34:02.056020 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.055995 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f952v" Sep 09 06:34:02.056333 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.056317 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-672fg\" (UniqueName: \"kubernetes.io/projected/2b9d7dfe-79ff-4abf-a090-7df31ea02020-kube-api-access-672fg\") pod \"network-check-target-mhgvf\" (UID: \"2b9d7dfe-79ff-4abf-a090-7df31ea02020\") " pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:34:02.155123 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.155099 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6vz4x\"" Sep 09 06:34:02.163142 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.163127 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:34:02.320138 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.320110 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f952v"] Sep 09 06:34:02.323255 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:34:02.323231 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bc0891_f9b1_469c_9337_c7832bb1d7ae.slice/crio-8317176054a8947ab67832db48826808f832586bac642635bc1a59314b190892 WatchSource:0}: Error finding container 8317176054a8947ab67832db48826808f832586bac642635bc1a59314b190892: Status 404 returned error can't find the container with id 8317176054a8947ab67832db48826808f832586bac642635bc1a59314b190892 Sep 09 06:34:02.404510 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.400273 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f952v" event={"ID":"e9bc0891-f9b1-469c-9337-c7832bb1d7ae","Type":"ContainerStarted","Data":"8317176054a8947ab67832db48826808f832586bac642635bc1a59314b190892"} Sep 09 06:34:02.426174 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.426150 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mhgvf"] Sep 09 06:34:02.428789 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:34:02.428765 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9d7dfe_79ff_4abf_a090_7df31ea02020.slice/crio-e64c64e6b307b546a0eb3cf765909caa308fd93518d3e3641acd553d4f86bd4f WatchSource:0}: Error finding container e64c64e6b307b546a0eb3cf765909caa308fd93518d3e3641acd553d4f86bd4f: Status 404 returned error can't find the container with id e64c64e6b307b546a0eb3cf765909caa308fd93518d3e3641acd553d4f86bd4f Sep 09 06:34:02.436283 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.436264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:34:02.436356 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:02.436291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:34:02.436407 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:02.436396 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 06:34:02.436438 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:02.436409 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69496f8585-qwx6b: secret "image-registry-tls" not found Sep 09 06:34:02.436469 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:02.436444 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls podName:2500df92-212c-4be0-8813-972d17dc1952 nodeName:}" failed. No retries permitted until 2026-09-09 06:34:34.4364321 +0000 UTC m=+98.761242534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls") pod "image-registry-69496f8585-qwx6b" (UID: "2500df92-212c-4be0-8813-972d17dc1952") : secret "image-registry-tls" not found Sep 09 06:34:02.436510 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:02.436395 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 06:34:02.436541 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:02.436516 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls podName:784894d0-7e1e-4611-9e85-c737ddbe3cad nodeName:}" failed. No retries permitted until 2026-09-09 06:34:34.436501978 +0000 UTC m=+98.761312415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls") pod "dns-default-d8257" (UID: "784894d0-7e1e-4611-9e85-c737ddbe3cad") : secret "dns-default-metrics-tls" not found Sep 09 06:34:03.405124 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:03.405053 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mhgvf" event={"ID":"2b9d7dfe-79ff-4abf-a090-7df31ea02020","Type":"ContainerStarted","Data":"e64c64e6b307b546a0eb3cf765909caa308fd93518d3e3641acd553d4f86bd4f"} Sep 09 06:34:07.417295 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:07.417258 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f952v" event={"ID":"e9bc0891-f9b1-469c-9337-c7832bb1d7ae","Type":"ContainerStarted","Data":"9290a2aafe4c745dc660991dd50f4467057dad6ead106a16d59babc7e61b31be"} Sep 09 06:34:07.418784 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:07.418764 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mhgvf" event={"ID":"2b9d7dfe-79ff-4abf-a090-7df31ea02020","Type":"ContainerStarted","Data":"eabb33fa605f949560434a1a0b56d1c46b81eb3bbdb4c5e14450ae61469ae947"} Sep 09 06:34:07.418875 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:07.418818 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:34:07.432406 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:07.432365 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f952v" podStartSLOduration=66.454450003 podStartE2EDuration="1m11.432352064s" totalImagesPullingTime="4.977902061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:34:07.432302767 +0000 UTC m=+71.757113222" watchObservedRunningTime="2026-09-09 06:34:07.432352064 +0000 UTC m=+71.757162520" Sep 09 06:34:34.445869 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:34.445713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:34:34.445869 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:34.445761 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:34:34.445869 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:34.445855 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 06:34:34.445869 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:34.445872 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 06:34:34.446513 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:34.445885 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69496f8585-qwx6b: secret "image-registry-tls" not found Sep 09 06:34:34.446513 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:34.445921 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls podName:784894d0-7e1e-4611-9e85-c737ddbe3cad nodeName:}" failed. No retries permitted until 2026-09-09 06:35:38.445904316 +0000 UTC m=+162.770714750 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls") pod "dns-default-d8257" (UID: "784894d0-7e1e-4611-9e85-c737ddbe3cad") : secret "dns-default-metrics-tls" not found Sep 09 06:34:34.446513 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:34:34.445942 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls podName:2500df92-212c-4be0-8813-972d17dc1952 nodeName:}" failed. No retries permitted until 2026-09-09 06:35:38.445928108 +0000 UTC m=+162.770738568 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls") pod "image-registry-69496f8585-qwx6b" (UID: "2500df92-212c-4be0-8813-972d17dc1952") : secret "image-registry-tls" not found Sep 09 06:34:38.422938 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:38.422910 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mhgvf" Sep 09 06:34:38.437059 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:34:38.437009 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mhgvf" podStartSLOduration=97.568062282 podStartE2EDuration="1m42.436953047s" totalImagesPullingTime="4.868890765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:34:07.453914037 +0000 UTC m=+71.778724494" watchObservedRunningTime="2026-09-09 06:34:38.436953047 +0000 UTC m=+102.761763500" Sep 09 06:35:05.847496 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:35:05.847452 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:35:05.848054 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:05.847577 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 06:35:05.848054 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:05.847648 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs podName:00c134a2-09bb-4d7a-abaa-8a81a37d2459 nodeName:}" failed. No retries permitted until 2026-09-09 06:37:07.847632212 +0000 UTC m=+252.172442645 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs") pod "network-metrics-daemon-p8hkq" (UID: "00c134a2-09bb-4d7a-abaa-8a81a37d2459") : secret "metrics-daemon-secret" not found Sep 09 06:35:33.559722 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:33.559679 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" podUID="2500df92-212c-4be0-8813-972d17dc1952" Sep 09 06:35:33.569938 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:33.569919 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d8257" podUID="784894d0-7e1e-4611-9e85-c737ddbe3cad" Sep 09 06:35:33.634354 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:35:33.634328 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:35:33.634472 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:35:33.634331 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d8257" Sep 09 06:35:34.161566 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:34.161528 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-p8hkq" podUID="00c134a2-09bb-4d7a-abaa-8a81a37d2459" Sep 09 06:35:38.462146 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:35:38.462108 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:35:38.462146 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:35:38.462144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:35:38.462574 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:38.462239 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 06:35:38.462574 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:38.462242 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 06:35:38.462574 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:38.462251 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69496f8585-qwx6b: secret "image-registry-tls" not found Sep 09 06:35:38.462574 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:38.462312 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls podName:2500df92-212c-4be0-8813-972d17dc1952 nodeName:}" failed. No retries permitted until 2026-09-09 06:37:40.462298 +0000 UTC m=+284.787108437 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls") pod "image-registry-69496f8585-qwx6b" (UID: "2500df92-212c-4be0-8813-972d17dc1952") : secret "image-registry-tls" not found Sep 09 06:35:38.462574 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:35:38.462324 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls podName:784894d0-7e1e-4611-9e85-c737ddbe3cad nodeName:}" failed. No retries permitted until 2026-09-09 06:37:40.462318491 +0000 UTC m=+284.787128924 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls") pod "dns-default-d8257" (UID: "784894d0-7e1e-4611-9e85-c737ddbe3cad") : secret "dns-default-metrics-tls" not found Sep 09 06:35:46.146100 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:35:46.144744 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:36:49.799659 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.799623 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66"] Sep 09 06:36:49.801749 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.801731 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:49.805325 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.805304 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 09 06:36:49.806555 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.806530 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 09 06:36:49.806674 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.806530 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 09 06:36:49.806674 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.806531 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 09 06:36:49.806674 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.806530 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5jlfg\"" Sep 09 06:36:49.806919 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.806902 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s"] Sep 09 06:36:49.809004 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.808986 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:49.811808 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.811788 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 09 06:36:49.811888 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.811813 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-k66qb\"" Sep 09 06:36:49.815131 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.815113 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 09 06:36:49.815408 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.815390 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 09 06:36:49.820715 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.820697 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66"] Sep 09 06:36:49.824198 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.824178 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s"] Sep 09 06:36:49.892493 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.892459 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7"] Sep 09 06:36:49.894278 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.894263 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7" Sep 09 06:36:49.896164 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.896142 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9"] Sep 09 06:36:49.897507 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.897491 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 09 06:36:49.897581 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.897523 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 09 06:36:49.897810 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.897795 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fpnc2\"" Sep 09 06:36:49.897886 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.897872 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bs7cf"] Sep 09 06:36:49.898060 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.898044 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:49.899518 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.899501 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-bs7cf" Sep 09 06:36:49.901389 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.901369 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 09 06:36:49.901704 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.901686 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 09 06:36:49.902099 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.902080 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 09 06:36:49.902099 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.902092 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 09 06:36:49.902223 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.902091 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mfwwj\"" Sep 09 06:36:49.902223 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.902209 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-88m8t\"" Sep 09 06:36:49.907022 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.907004 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7"] Sep 09 06:36:49.909648 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.909626 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bs7cf"] Sep 09 06:36:49.919644 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.919623 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9"] Sep 09 06:36:49.936094 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.936060 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ace615ac-3fae-4d50-9e0b-e5b3d149f51f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4h66\" (UID: \"ace615ac-3fae-4d50-9e0b-e5b3d149f51f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:49.936094 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.936093 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wnnc\" (UniqueName: \"kubernetes.io/projected/ace615ac-3fae-4d50-9e0b-e5b3d149f51f-kube-api-access-8wnnc\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4h66\" (UID: \"ace615ac-3fae-4d50-9e0b-e5b3d149f51f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:49.936283 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.936120 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:49.936283 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.936219 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swblj\" (UniqueName: \"kubernetes.io/projected/77cf1306-5823-4ed6-826c-2fad59e20080-kube-api-access-swblj\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:49.936283 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.936278 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7qp\" (UniqueName: \"kubernetes.io/projected/07921b1d-7654-4c49-81cb-d0716848377b-kube-api-access-5w7qp\") pod \"volume-data-source-validator-695c895bd6-jx9q7\" (UID: \"07921b1d-7654-4c49-81cb-d0716848377b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7" Sep 09 06:36:49.936417 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:49.936331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ace615ac-3fae-4d50-9e0b-e5b3d149f51f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4h66\" (UID: \"ace615ac-3fae-4d50-9e0b-e5b3d149f51f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:50.037620 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.037588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ace615ac-3fae-4d50-9e0b-e5b3d149f51f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4h66\" (UID: \"ace615ac-3fae-4d50-9e0b-e5b3d149f51f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:50.037620 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.037622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ace615ac-3fae-4d50-9e0b-e5b3d149f51f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4h66\" (UID: \"ace615ac-3fae-4d50-9e0b-e5b3d149f51f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:50.037620 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.037639 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wnnc\" (UniqueName: \"kubernetes.io/projected/ace615ac-3fae-4d50-9e0b-e5b3d149f51f-kube-api-access-8wnnc\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4h66\" (UID: \"ace615ac-3fae-4d50-9e0b-e5b3d149f51f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:50.037948 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.037656 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:50.037948 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.037679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af365614-5695-450a-a777-f25e24f4d564-serving-cert\") pod \"service-ca-operator-859f94c4b-knzv9\" (UID: \"af365614-5695-450a-a777-f25e24f4d564\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.037948 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.037701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af365614-5695-450a-a777-f25e24f4d564-config\") pod \"service-ca-operator-859f94c4b-knzv9\" (UID: \"af365614-5695-450a-a777-f25e24f4d564\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.037948 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.037750 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swblj\" (UniqueName: \"kubernetes.io/projected/77cf1306-5823-4ed6-826c-2fad59e20080-kube-api-access-swblj\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:50.037948 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:50.037807 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 06:36:50.037948 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:50.037910 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls podName:77cf1306-5823-4ed6-826c-2fad59e20080 nodeName:}" failed. No retries permitted until 2026-09-09 06:36:50.537889003 +0000 UTC m=+234.862699441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hlb8s" (UID: "77cf1306-5823-4ed6-826c-2fad59e20080") : secret "samples-operator-tls" not found Sep 09 06:36:50.038256 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.037806 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7qp\" (UniqueName: \"kubernetes.io/projected/07921b1d-7654-4c49-81cb-d0716848377b-kube-api-access-5w7qp\") pod \"volume-data-source-validator-695c895bd6-jx9q7\" (UID: \"07921b1d-7654-4c49-81cb-d0716848377b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7" Sep 09 06:36:50.038307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.038259 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkr7t\" (UniqueName: \"kubernetes.io/projected/af365614-5695-450a-a777-f25e24f4d564-kube-api-access-qkr7t\") pod \"service-ca-operator-859f94c4b-knzv9\" (UID: \"af365614-5695-450a-a777-f25e24f4d564\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.038307 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.038289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmv4n\" (UniqueName: \"kubernetes.io/projected/f45f5765-d893-4325-a7d7-83dd1ad5a627-kube-api-access-nmv4n\") pod \"network-check-source-685456655f-bs7cf\" (UID: \"f45f5765-d893-4325-a7d7-83dd1ad5a627\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bs7cf" Sep 09 06:36:50.038399 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.038381 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ace615ac-3fae-4d50-9e0b-e5b3d149f51f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4h66\" (UID: \"ace615ac-3fae-4d50-9e0b-e5b3d149f51f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:50.040106 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.040090 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ace615ac-3fae-4d50-9e0b-e5b3d149f51f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4h66\" (UID: \"ace615ac-3fae-4d50-9e0b-e5b3d149f51f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:50.051389 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.051313 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7qp\" (UniqueName: \"kubernetes.io/projected/07921b1d-7654-4c49-81cb-d0716848377b-kube-api-access-5w7qp\") pod \"volume-data-source-validator-695c895bd6-jx9q7\" (UID: \"07921b1d-7654-4c49-81cb-d0716848377b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7" Sep 09 06:36:50.051389 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.051343 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swblj\" (UniqueName: \"kubernetes.io/projected/77cf1306-5823-4ed6-826c-2fad59e20080-kube-api-access-swblj\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:50.053059 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.053037 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wnnc\" (UniqueName: \"kubernetes.io/projected/ace615ac-3fae-4d50-9e0b-e5b3d149f51f-kube-api-access-8wnnc\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4h66\" (UID: \"ace615ac-3fae-4d50-9e0b-e5b3d149f51f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:50.111165 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.111107 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" Sep 09 06:36:50.139212 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.139175 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af365614-5695-450a-a777-f25e24f4d564-serving-cert\") pod \"service-ca-operator-859f94c4b-knzv9\" (UID: \"af365614-5695-450a-a777-f25e24f4d564\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.139308 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.139218 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af365614-5695-450a-a777-f25e24f4d564-config\") pod \"service-ca-operator-859f94c4b-knzv9\" (UID: \"af365614-5695-450a-a777-f25e24f4d564\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.139352 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.139303 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkr7t\" (UniqueName: \"kubernetes.io/projected/af365614-5695-450a-a777-f25e24f4d564-kube-api-access-qkr7t\") pod \"service-ca-operator-859f94c4b-knzv9\" (UID: \"af365614-5695-450a-a777-f25e24f4d564\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.139352 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.139330 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmv4n\" (UniqueName: \"kubernetes.io/projected/f45f5765-d893-4325-a7d7-83dd1ad5a627-kube-api-access-nmv4n\") pod \"network-check-source-685456655f-bs7cf\" (UID: \"f45f5765-d893-4325-a7d7-83dd1ad5a627\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bs7cf" Sep 09 06:36:50.139845 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.139785 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af365614-5695-450a-a777-f25e24f4d564-config\") pod \"service-ca-operator-859f94c4b-knzv9\" (UID: \"af365614-5695-450a-a777-f25e24f4d564\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.141465 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.141446 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af365614-5695-450a-a777-f25e24f4d564-serving-cert\") pod \"service-ca-operator-859f94c4b-knzv9\" (UID: \"af365614-5695-450a-a777-f25e24f4d564\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.148316 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.148208 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkr7t\" (UniqueName: \"kubernetes.io/projected/af365614-5695-450a-a777-f25e24f4d564-kube-api-access-qkr7t\") pod \"service-ca-operator-859f94c4b-knzv9\" (UID: \"af365614-5695-450a-a777-f25e24f4d564\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.148316 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.148252 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmv4n\" (UniqueName: \"kubernetes.io/projected/f45f5765-d893-4325-a7d7-83dd1ad5a627-kube-api-access-nmv4n\") pod \"network-check-source-685456655f-bs7cf\" (UID: \"f45f5765-d893-4325-a7d7-83dd1ad5a627\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bs7cf" Sep 09 06:36:50.204434 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.204402 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7" Sep 09 06:36:50.211834 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.211811 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" Sep 09 06:36:50.216591 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.216560 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-bs7cf" Sep 09 06:36:50.380610 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.380520 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66"] Sep 09 06:36:50.383034 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:36:50.383008 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace615ac_3fae_4d50_9e0b_e5b3d149f51f.slice/crio-1c0e9c2d70dfb6bc11171dca0a3d989b8ce45f74c3a74b510f14b80edc478277 WatchSource:0}: Error finding container 1c0e9c2d70dfb6bc11171dca0a3d989b8ce45f74c3a74b510f14b80edc478277: Status 404 returned error can't find the container with id 1c0e9c2d70dfb6bc11171dca0a3d989b8ce45f74c3a74b510f14b80edc478277 Sep 09 06:36:50.475306 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.472383 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7"] Sep 09 06:36:50.476355 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:36:50.476324 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07921b1d_7654_4c49_81cb_d0716848377b.slice/crio-3e3084a7c65d25babeb868131fec90014df3cba842ed827022642ece9eb13aa6 WatchSource:0}: Error finding container 3e3084a7c65d25babeb868131fec90014df3cba842ed827022642ece9eb13aa6: Status 404 returned error can't find the container with id 3e3084a7c65d25babeb868131fec90014df3cba842ed827022642ece9eb13aa6 Sep 09 06:36:50.497454 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.497431 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bs7cf"] Sep 09 06:36:50.500342 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:36:50.500314 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45f5765_d893_4325_a7d7_83dd1ad5a627.slice/crio-e18acc8c4992f06f7eff72ee9b58db2e992911f4139655af6ce89a046761bcc1 WatchSource:0}: Error finding container e18acc8c4992f06f7eff72ee9b58db2e992911f4139655af6ce89a046761bcc1: Status 404 returned error can't find the container with id e18acc8c4992f06f7eff72ee9b58db2e992911f4139655af6ce89a046761bcc1 Sep 09 06:36:50.500575 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.500557 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9"] Sep 09 06:36:50.503001 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:36:50.502983 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf365614_5695_450a_a777_f25e24f4d564.slice/crio-812ad0041263510308ec20e26af234acc5b3c32093712053d39d128436026c6f WatchSource:0}: Error finding container 812ad0041263510308ec20e26af234acc5b3c32093712053d39d128436026c6f: Status 404 returned error can't find the container with id 812ad0041263510308ec20e26af234acc5b3c32093712053d39d128436026c6f Sep 09 06:36:50.542067 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.542047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:50.542188 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:50.542151 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 06:36:50.542230 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:50.542202 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls podName:77cf1306-5823-4ed6-826c-2fad59e20080 nodeName:}" failed. No retries permitted until 2026-09-09 06:36:51.542189359 +0000 UTC m=+235.866999792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hlb8s" (UID: "77cf1306-5823-4ed6-826c-2fad59e20080") : secret "samples-operator-tls" not found Sep 09 06:36:50.818633 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.818556 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" event={"ID":"af365614-5695-450a-a777-f25e24f4d564","Type":"ContainerStarted","Data":"812ad0041263510308ec20e26af234acc5b3c32093712053d39d128436026c6f"} Sep 09 06:36:50.819769 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.819744 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7" event={"ID":"07921b1d-7654-4c49-81cb-d0716848377b","Type":"ContainerStarted","Data":"3e3084a7c65d25babeb868131fec90014df3cba842ed827022642ece9eb13aa6"} Sep 09 06:36:50.820816 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.820793 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" event={"ID":"ace615ac-3fae-4d50-9e0b-e5b3d149f51f","Type":"ContainerStarted","Data":"1c0e9c2d70dfb6bc11171dca0a3d989b8ce45f74c3a74b510f14b80edc478277"} Sep 09 06:36:50.822082 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.822064 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-bs7cf" event={"ID":"f45f5765-d893-4325-a7d7-83dd1ad5a627","Type":"ContainerStarted","Data":"52981f5b027dc5d66eef7051089762b54ae5f7f5a8d249e0dad8cd2e3a21c45a"} Sep 09 06:36:50.822169 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.822096 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-bs7cf" event={"ID":"f45f5765-d893-4325-a7d7-83dd1ad5a627","Type":"ContainerStarted","Data":"e18acc8c4992f06f7eff72ee9b58db2e992911f4139655af6ce89a046761bcc1"} Sep 09 06:36:50.848644 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:50.848606 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-bs7cf" podStartSLOduration=1.848593695 podStartE2EDuration="1.848593695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:36:50.847874038 +0000 UTC m=+235.172684495" watchObservedRunningTime="2026-09-09 06:36:50.848593695 +0000 UTC m=+235.173404142" Sep 09 06:36:51.549628 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:51.549584 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:51.549824 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:51.549732 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 06:36:51.549824 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:51.549793 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls podName:77cf1306-5823-4ed6-826c-2fad59e20080 nodeName:}" failed. No retries permitted until 2026-09-09 06:36:53.549773952 +0000 UTC m=+237.874584391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hlb8s" (UID: "77cf1306-5823-4ed6-826c-2fad59e20080") : secret "samples-operator-tls" not found Sep 09 06:36:53.567910 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:53.567871 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:53.568379 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:53.568018 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 06:36:53.568379 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:53.568092 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls podName:77cf1306-5823-4ed6-826c-2fad59e20080 nodeName:}" failed. No retries permitted until 2026-09-09 06:36:57.568072118 +0000 UTC m=+241.892882569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hlb8s" (UID: "77cf1306-5823-4ed6-826c-2fad59e20080") : secret "samples-operator-tls" not found Sep 09 06:36:53.833202 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:53.833114 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" event={"ID":"af365614-5695-450a-a777-f25e24f4d564","Type":"ContainerStarted","Data":"96a193646bc17d0ea8a4681fc9daca41c2e1202916061c64f6832dc9966060de"} Sep 09 06:36:53.834830 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:53.834797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7" event={"ID":"07921b1d-7654-4c49-81cb-d0716848377b","Type":"ContainerStarted","Data":"bd1928816d2d445774ad8064d211f54d553774f168f82220aab8cf55d38981a4"} Sep 09 06:36:53.836410 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:53.836392 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" event={"ID":"ace615ac-3fae-4d50-9e0b-e5b3d149f51f","Type":"ContainerStarted","Data":"fe170b8612d0818ac9b126af30fbc4604d5208a54f3e784d4b875791be7fa743"} Sep 09 06:36:53.871459 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:53.871389 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" podStartSLOduration=2.308439936 podStartE2EDuration="4.871370018s" totalImagesPullingTime="2.562930082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:36:53.87082763 +0000 UTC m=+238.195638099" watchObservedRunningTime="2026-09-09 06:36:53.871370018 +0000 UTC m=+238.196180475" Sep 09 06:36:53.871982 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:53.871917 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" podStartSLOduration=2.4275513699999998 podStartE2EDuration="4.871902186s" totalImagesPullingTime="2.444350816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:36:53.850641769 +0000 UTC m=+238.175452224" watchObservedRunningTime="2026-09-09 06:36:53.871902186 +0000 UTC m=+238.196712649" Sep 09 06:36:53.892504 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:53.892461 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jx9q7" podStartSLOduration=2.429407887 podStartE2EDuration="4.892449226s" totalImagesPullingTime="2.463041339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:36:53.891879904 +0000 UTC m=+238.216690361" watchObservedRunningTime="2026-09-09 06:36:53.892449226 +0000 UTC m=+238.217259691" Sep 09 06:36:54.294982 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.294933 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bl97r"] Sep 09 06:36:54.297291 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.297269 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.300038 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.300016 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 06:36:54.300428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.300414 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 06:36:54.301494 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.301339 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 06:36:54.301494 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.301350 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pcccg\"" Sep 09 06:36:54.301494 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.301357 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 06:36:54.327921 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.325858 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bl97r"] Sep 09 06:36:54.374633 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.374606 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-data-volume\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.374748 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.374652 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctrtb\" (UniqueName: \"kubernetes.io/projected/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-api-access-ctrtb\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.374748 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.374670 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.374824 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.374746 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.374824 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.374779 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-crio-socket\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.475892 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.475855 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctrtb\" (UniqueName: \"kubernetes.io/projected/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-api-access-ctrtb\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.476025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.475897 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.476025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.475986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.476132 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.476022 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-crio-socket\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.476132 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:54.476055 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 06:36:54.476132 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:54.476073 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 06:36:54.476132 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.476088 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-data-volume\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.476132 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:54.476132 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:36:54.976111994 +0000 UTC m=+239.300922442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : secret "insights-runtime-extractor-tls" not found Sep 09 06:36:54.476292 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:54.476159 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:36:54.976140728 +0000 UTC m=+239.300951179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : configmap "kube-rbac-proxy" not found Sep 09 06:36:54.476292 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.476173 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-crio-socket\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.476439 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.476423 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-data-volume\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.485281 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.485259 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctrtb\" (UniqueName: \"kubernetes.io/projected/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-api-access-ctrtb\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.979978 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.979940 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.980577 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.980018 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:54.980577 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:54.980099 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 06:36:54.980577 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:54.980176 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 06:36:54.980577 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:54.980182 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:36:55.980161786 +0000 UTC m=+240.304972240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : secret "insights-runtime-extractor-tls" not found Sep 09 06:36:54.980577 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:54.980362 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:36:55.980328217 +0000 UTC m=+240.305138651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : configmap "kube-rbac-proxy" not found Sep 09 06:36:54.988102 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.988081 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs"] Sep 09 06:36:54.990657 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.990641 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs" Sep 09 06:36:54.993609 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.993593 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9swzd\"" Sep 09 06:36:54.994705 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.994683 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 09 06:36:54.994787 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:54.994693 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 09 06:36:55.002075 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:55.002056 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs"] Sep 09 06:36:55.081054 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:55.081021 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbftd\" (UniqueName: \"kubernetes.io/projected/8d403b3f-a880-428c-bd1b-e39c557c9e97-kube-api-access-vbftd\") pod \"migrator-fdbb8fdff-bz9hs\" (UID: \"8d403b3f-a880-428c-bd1b-e39c557c9e97\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs" Sep 09 06:36:55.181850 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:55.181820 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbftd\" (UniqueName: \"kubernetes.io/projected/8d403b3f-a880-428c-bd1b-e39c557c9e97-kube-api-access-vbftd\") pod \"migrator-fdbb8fdff-bz9hs\" (UID: \"8d403b3f-a880-428c-bd1b-e39c557c9e97\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs" Sep 09 06:36:55.192943 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:55.192921 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbftd\" (UniqueName: \"kubernetes.io/projected/8d403b3f-a880-428c-bd1b-e39c557c9e97-kube-api-access-vbftd\") pod \"migrator-fdbb8fdff-bz9hs\" (UID: \"8d403b3f-a880-428c-bd1b-e39c557c9e97\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs" Sep 09 06:36:55.299684 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:55.299609 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs" Sep 09 06:36:55.594974 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:55.594831 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs"] Sep 09 06:36:55.597366 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:36:55.597328 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d403b3f_a880_428c_bd1b_e39c557c9e97.slice/crio-e155ad5634a35356cd89cef36b6913fc44232179e7d47b46f5af93ad3c0dffb7 WatchSource:0}: Error finding container e155ad5634a35356cd89cef36b6913fc44232179e7d47b46f5af93ad3c0dffb7: Status 404 returned error can't find the container with id e155ad5634a35356cd89cef36b6913fc44232179e7d47b46f5af93ad3c0dffb7 Sep 09 06:36:55.843694 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:55.843656 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs" event={"ID":"8d403b3f-a880-428c-bd1b-e39c557c9e97","Type":"ContainerStarted","Data":"e155ad5634a35356cd89cef36b6913fc44232179e7d47b46f5af93ad3c0dffb7"} Sep 09 06:36:55.986458 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:55.986427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:55.986880 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:55.986477 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:55.986880 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:55.986563 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 06:36:55.986880 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:55.986607 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 06:36:55.986880 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:55.986613 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:36:57.986600136 +0000 UTC m=+242.311410571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : configmap "kube-rbac-proxy" not found Sep 09 06:36:55.986880 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:55.986686 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:36:57.986665909 +0000 UTC m=+242.311476349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : secret "insights-runtime-extractor-tls" not found Sep 09 06:36:56.222302 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:56.222274 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sv5qm_3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec/dns-node-resolver/0.log" Sep 09 06:36:57.222752 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:57.222725 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-plfqc_eb076d84-0bb2-474a-a6d7-c83e1c98904c/node-ca/0.log" Sep 09 06:36:57.597085 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:57.597057 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:36:57.597265 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:57.597230 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 06:36:57.597331 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:57.597315 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls podName:77cf1306-5823-4ed6-826c-2fad59e20080 nodeName:}" failed. No retries permitted until 2026-09-09 06:37:05.597292751 +0000 UTC m=+249.922103205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hlb8s" (UID: "77cf1306-5823-4ed6-826c-2fad59e20080") : secret "samples-operator-tls" not found Sep 09 06:36:57.851279 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:57.851190 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs" event={"ID":"8d403b3f-a880-428c-bd1b-e39c557c9e97","Type":"ContainerStarted","Data":"7d32de1ff2cc5fd031619e2aa5fc2f5d8250bad50667ca2e95e74c16590448fa"} Sep 09 06:36:57.851279 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:57.851269 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs" event={"ID":"8d403b3f-a880-428c-bd1b-e39c557c9e97","Type":"ContainerStarted","Data":"757cfe313edea43c64f9964224a1f71a50a79e47f68f7f3d4c0c36a79350db47"} Sep 09 06:36:57.876042 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:57.875992 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-bz9hs" podStartSLOduration=2.6369965029999998 podStartE2EDuration="3.875976741s" totalImagesPullingTime="1.238980238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:36:57.87536312 +0000 UTC m=+242.200173595" watchObservedRunningTime="2026-09-09 06:36:57.875976741 +0000 UTC m=+242.200787188" Sep 09 06:36:57.999780 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:57.999729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:57.999922 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:57.999798 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:36:57.999922 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:57.999886 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 06:36:57.999922 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:57.999910 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 06:36:58.000073 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:57.999984 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:37:01.999943586 +0000 UTC m=+246.324754033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : secret "insights-runtime-extractor-tls" not found Sep 09 06:36:58.000073 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:36:58.000005 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:37:01.999995365 +0000 UTC m=+246.324805799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : configmap "kube-rbac-proxy" not found Sep 09 06:36:58.021757 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:36:58.021732 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8t8b_02718a56-6748-4bb7-9cef-01d5fe02132a/serve-healthcheck-canary/0.log" Sep 09 06:37:02.033095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:02.033058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:37:02.033533 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:02.033106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:37:02.033533 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:02.033178 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 06:37:02.033533 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:02.033219 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 06:37:02.033533 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:02.033227 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:37:10.033213601 +0000 UTC m=+254.358024035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : configmap "kube-rbac-proxy" not found Sep 09 06:37:02.033533 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:02.033289 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:37:10.033272586 +0000 UTC m=+254.358083020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : secret "insights-runtime-extractor-tls" not found Sep 09 06:37:05.662422 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:05.662383 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:37:05.664817 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:05.664794 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77cf1306-5823-4ed6-826c-2fad59e20080-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hlb8s\" (UID: \"77cf1306-5823-4ed6-826c-2fad59e20080\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:37:05.720990 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:05.720948 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-k66qb\"" Sep 09 06:37:05.728203 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:05.728187 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" Sep 09 06:37:06.000210 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:06.000145 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s"] Sep 09 06:37:06.881503 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:06.881462 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" event={"ID":"77cf1306-5823-4ed6-826c-2fad59e20080","Type":"ContainerStarted","Data":"d34d93e3ecc2ada26ca6cbdf69b00fd622e1a9529fd227121fa97733762a527a"} Sep 09 06:37:07.882754 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:07.882669 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:37:07.885383 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:07.885357 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c134a2-09bb-4d7a-abaa-8a81a37d2459-metrics-certs\") pod \"network-metrics-daemon-p8hkq\" (UID: \"00c134a2-09bb-4d7a-abaa-8a81a37d2459\") " pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:37:07.886386 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:07.886356 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" event={"ID":"77cf1306-5823-4ed6-826c-2fad59e20080","Type":"ContainerStarted","Data":"d9715b42eb970c2a80560b8c44dabfc0040640f0282eef2d6f0ab6afa432dfe5"} Sep 09 06:37:07.886500 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:07.886424 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" event={"ID":"77cf1306-5823-4ed6-826c-2fad59e20080","Type":"ContainerStarted","Data":"25b73b6f89c7b9424d75677d381533d04a7b6d98540dedb1b8100884328d98bd"} Sep 09 06:37:07.909660 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:07.909624 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hlb8s" podStartSLOduration=17.335930238 podStartE2EDuration="18.90961056s" totalImagesPullingTime="1.573680322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:07.907916699 +0000 UTC m=+252.232727156" watchObservedRunningTime="2026-09-09 06:37:07.90961056 +0000 UTC m=+252.234421015" Sep 09 06:37:08.049080 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:08.049058 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gdzz9\"" Sep 09 06:37:08.056452 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:08.056439 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8hkq" Sep 09 06:37:08.335440 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:08.334651 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8hkq"] Sep 09 06:37:08.340471 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:08.340437 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c134a2_09bb_4d7a_abaa_8a81a37d2459.slice/crio-17ef7498b782bf0ed92fcb3d4c980a34c2aeac39027d7962ef7d265b63bce0af WatchSource:0}: Error finding container 17ef7498b782bf0ed92fcb3d4c980a34c2aeac39027d7962ef7d265b63bce0af: Status 404 returned error can't find the container with id 17ef7498b782bf0ed92fcb3d4c980a34c2aeac39027d7962ef7d265b63bce0af Sep 09 06:37:08.890408 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:08.890374 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8hkq" event={"ID":"00c134a2-09bb-4d7a-abaa-8a81a37d2459","Type":"ContainerStarted","Data":"17ef7498b782bf0ed92fcb3d4c980a34c2aeac39027d7962ef7d265b63bce0af"} Sep 09 06:37:09.895180 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:09.895147 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8hkq" event={"ID":"00c134a2-09bb-4d7a-abaa-8a81a37d2459","Type":"ContainerStarted","Data":"39ea56142232cab6c339fccf91ce4b4066a8fbe5b9bea64971593a0ecd2d1788"} Sep 09 06:37:09.895544 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:09.895203 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8hkq" event={"ID":"00c134a2-09bb-4d7a-abaa-8a81a37d2459","Type":"ContainerStarted","Data":"0f8a0b7d9de69631677ee5f2900d84b9a084cc50d1b2cea057a66db3f9e3eb4a"} Sep 09 06:37:10.101185 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:10.101156 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:37:10.101346 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:10.101201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:37:10.101346 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:10.101317 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 06:37:10.101414 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:10.101407 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls podName:6b6445dd-fad0-49e5-aeaa-98bd2849a12c nodeName:}" failed. No retries permitted until 2026-09-09 06:37:26.101385926 +0000 UTC m=+270.426196375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bl97r" (UID: "6b6445dd-fad0-49e5-aeaa-98bd2849a12c") : secret "insights-runtime-extractor-tls" not found Sep 09 06:37:10.101662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:10.101646 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:37:25.043491 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.043398 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p8hkq" podStartSLOduration=267.909247935 podStartE2EDuration="4m29.043381403s" totalImagesPullingTime="1.134133468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:09.925546681 +0000 UTC m=+254.250357137" watchObservedRunningTime="2026-09-09 06:37:25.043381403 +0000 UTC m=+269.368191859" Sep 09 06:37:25.044265 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.044249 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx"] Sep 09 06:37:25.047314 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.047295 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" Sep 09 06:37:25.052665 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.052644 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 06:37:25.052758 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.052669 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9bn5v\"" Sep 09 06:37:25.062756 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.062728 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx"] Sep 09 06:37:25.211490 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.211464 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8fbc92c-020b-4485-9726-25672578965f-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8krnx\" (UID: \"f8fbc92c-020b-4485-9726-25672578965f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" Sep 09 06:37:25.312330 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.312250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8fbc92c-020b-4485-9726-25672578965f-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8krnx\" (UID: \"f8fbc92c-020b-4485-9726-25672578965f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" Sep 09 06:37:25.312450 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:25.312378 2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 09 06:37:25.312450 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:25.312438 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8fbc92c-020b-4485-9726-25672578965f-tls-certificates podName:f8fbc92c-020b-4485-9726-25672578965f nodeName:}" failed. No retries permitted until 2026-09-09 06:37:25.812425512 +0000 UTC m=+270.137235946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f8fbc92c-020b-4485-9726-25672578965f-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-8krnx" (UID: "f8fbc92c-020b-4485-9726-25672578965f") : secret "prometheus-operator-admission-webhook-tls" not found Sep 09 06:37:25.815883 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.815848 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8fbc92c-020b-4485-9726-25672578965f-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8krnx\" (UID: \"f8fbc92c-020b-4485-9726-25672578965f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" Sep 09 06:37:25.818529 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.818508 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8fbc92c-020b-4485-9726-25672578965f-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8krnx\" (UID: \"f8fbc92c-020b-4485-9726-25672578965f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" Sep 09 06:37:25.955398 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:25.955367 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" Sep 09 06:37:26.118131 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:26.118100 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:37:26.120420 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:26.120394 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b6445dd-fad0-49e5-aeaa-98bd2849a12c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bl97r\" (UID: \"6b6445dd-fad0-49e5-aeaa-98bd2849a12c\") " pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:37:26.241566 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:26.241538 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx"] Sep 09 06:37:26.244517 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:26.244490 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fbc92c_020b_4485_9726_25672578965f.slice/crio-be6d868e4fe96de6b3db2e8cec16a27e3333356a70367ff9b7d7d2a5225e9f7d WatchSource:0}: Error finding container be6d868e4fe96de6b3db2e8cec16a27e3333356a70367ff9b7d7d2a5225e9f7d: Status 404 returned error can't find the container with id be6d868e4fe96de6b3db2e8cec16a27e3333356a70367ff9b7d7d2a5225e9f7d Sep 09 06:37:26.412521 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:26.412451 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pcccg\"" Sep 09 06:37:26.419644 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:26.419632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bl97r" Sep 09 06:37:26.704539 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:26.704456 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bl97r"] Sep 09 06:37:26.708217 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:26.708197 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6445dd_fad0_49e5_aeaa_98bd2849a12c.slice/crio-a907865c78b6992474f365981a2162732a03bee95a1991f1441bbf0d56752083 WatchSource:0}: Error finding container a907865c78b6992474f365981a2162732a03bee95a1991f1441bbf0d56752083: Status 404 returned error can't find the container with id a907865c78b6992474f365981a2162732a03bee95a1991f1441bbf0d56752083 Sep 09 06:37:26.953441 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:26.953391 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" event={"ID":"f8fbc92c-020b-4485-9726-25672578965f","Type":"ContainerStarted","Data":"be6d868e4fe96de6b3db2e8cec16a27e3333356a70367ff9b7d7d2a5225e9f7d"} Sep 09 06:37:26.955256 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:26.955181 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bl97r" event={"ID":"6b6445dd-fad0-49e5-aeaa-98bd2849a12c","Type":"ContainerStarted","Data":"fcb987262dd42308487723049ccaed81a51a15c8a63ecb24092d9b6c50a1dfa9"} Sep 09 06:37:26.955256 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:26.955241 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bl97r" event={"ID":"6b6445dd-fad0-49e5-aeaa-98bd2849a12c","Type":"ContainerStarted","Data":"a907865c78b6992474f365981a2162732a03bee95a1991f1441bbf0d56752083"} Sep 09 06:37:27.962452 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:27.962378 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bl97r" event={"ID":"6b6445dd-fad0-49e5-aeaa-98bd2849a12c","Type":"ContainerStarted","Data":"882db3065e4085cdc669cf741d94a269acc4d67d609f2f7362c57ad24545ec13"} Sep 09 06:37:27.963874 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:27.963850 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" event={"ID":"f8fbc92c-020b-4485-9726-25672578965f","Type":"ContainerStarted","Data":"f358d64ec0b85a78fa0f6d2475e32d2467696c07f16b5073749fc70cdbbcbbbf"} Sep 09 06:37:27.964172 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:27.964148 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" Sep 09 06:37:27.968636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:27.968610 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" Sep 09 06:37:27.984215 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:27.984175 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8krnx" podStartSLOduration=1.855215257 podStartE2EDuration="2.984162576s" totalImagesPullingTime="1.128947319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:27.98246075 +0000 UTC m=+272.307271206" watchObservedRunningTime="2026-09-09 06:37:27.984162576 +0000 UTC m=+272.308973080" Sep 09 06:37:28.969102 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:28.969066 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bl97r" event={"ID":"6b6445dd-fad0-49e5-aeaa-98bd2849a12c","Type":"ContainerStarted","Data":"61d087258795873c73b67c2dc067ab829096d02f909b0659babbc9443128d639"} Sep 09 06:37:29.023827 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.023785 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bl97r" podStartSLOduration=32.988316458 podStartE2EDuration="35.02377012s" totalImagesPullingTime="2.035453662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:36:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:29.022698027 +0000 UTC m=+273.347508483" watchObservedRunningTime="2026-09-09 06:37:29.02377012 +0000 UTC m=+273.348580576" Sep 09 06:37:29.231201 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.231139 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-k5x9g"] Sep 09 06:37:29.234315 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.234296 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.239114 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.239095 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 06:37:29.239699 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.239682 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 06:37:29.240353 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.240286 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 09 06:37:29.240425 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.240390 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 09 06:37:29.240480 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.240437 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nql99\"" Sep 09 06:37:29.240673 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.240646 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 06:37:29.310696 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.310672 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-k5x9g"] Sep 09 06:37:29.340855 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.340834 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3a84eba-7709-40c9-987a-f1dba4a79cd8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.340987 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.340865 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a84eba-7709-40c9-987a-f1dba4a79cd8-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.340987 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.340895 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gq8f\" (UniqueName: \"kubernetes.io/projected/f3a84eba-7709-40c9-987a-f1dba4a79cd8-kube-api-access-7gq8f\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.340987 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.340919 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3a84eba-7709-40c9-987a-f1dba4a79cd8-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.441269 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.441240 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a84eba-7709-40c9-987a-f1dba4a79cd8-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.441413 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.441278 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq8f\" (UniqueName: \"kubernetes.io/projected/f3a84eba-7709-40c9-987a-f1dba4a79cd8-kube-api-access-7gq8f\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.441413 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.441305 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3a84eba-7709-40c9-987a-f1dba4a79cd8-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.441413 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.441356 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3a84eba-7709-40c9-987a-f1dba4a79cd8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.441920 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.441895 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a84eba-7709-40c9-987a-f1dba4a79cd8-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.443877 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.443853 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3a84eba-7709-40c9-987a-f1dba4a79cd8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.443949 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.443893 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3a84eba-7709-40c9-987a-f1dba4a79cd8-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.451783 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.451761 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq8f\" (UniqueName: \"kubernetes.io/projected/f3a84eba-7709-40c9-987a-f1dba4a79cd8-kube-api-access-7gq8f\") pod \"prometheus-operator-6f4946db56-k5x9g\" (UID: \"f3a84eba-7709-40c9-987a-f1dba4a79cd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.543015 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.542954 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" Sep 09 06:37:29.831146 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.831116 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-k5x9g"] Sep 09 06:37:29.831940 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:29.831921 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a84eba_7709_40c9_987a_f1dba4a79cd8.slice/crio-bc43e52d9d7f7b7bcecc5e3bcd09719e82082fdb960d921b95387d78948a920c WatchSource:0}: Error finding container bc43e52d9d7f7b7bcecc5e3bcd09719e82082fdb960d921b95387d78948a920c: Status 404 returned error can't find the container with id bc43e52d9d7f7b7bcecc5e3bcd09719e82082fdb960d921b95387d78948a920c Sep 09 06:37:29.973124 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:29.973093 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" event={"ID":"f3a84eba-7709-40c9-987a-f1dba4a79cd8","Type":"ContainerStarted","Data":"bc43e52d9d7f7b7bcecc5e3bcd09719e82082fdb960d921b95387d78948a920c"} Sep 09 06:37:31.983048 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:31.983014 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" event={"ID":"f3a84eba-7709-40c9-987a-f1dba4a79cd8","Type":"ContainerStarted","Data":"a9bc258e7b6a7997b73f5608a53b2174f38876e6ca658554c6a25d698845d2c1"} Sep 09 06:37:31.983410 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:31.983062 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" event={"ID":"f3a84eba-7709-40c9-987a-f1dba4a79cd8","Type":"ContainerStarted","Data":"d0bce9df51db361469c65a29856466f11ddb5f5a4c35123ce8bbc0b191a28eea"} Sep 09 06:37:32.025402 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:32.025359 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-k5x9g" podStartSLOduration=1.62659502 podStartE2EDuration="3.025340666s" totalImagesPullingTime="1.398745646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:32.024742458 +0000 UTC m=+276.349552914" watchObservedRunningTime="2026-09-09 06:37:32.025340666 +0000 UTC m=+276.350151123" Sep 09 06:37:33.731724 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.731693 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr"] Sep 09 06:37:33.735034 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.735019 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.741153 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.740934 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 09 06:37:33.741153 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.740978 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5xdvq\"" Sep 09 06:37:33.741153 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.740937 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 09 06:37:33.763700 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.763668 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw"] Sep 09 06:37:33.767150 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.767125 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr"] Sep 09 06:37:33.767260 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.767247 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.769706 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.769684 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c35f04b-688c-4f62-b55f-d757aeb29653-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.769806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.769711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9vcf\" (UniqueName: \"kubernetes.io/projected/41d8bb2e-ae23-407a-9218-019bfc151181-kube-api-access-p9vcf\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.769806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.769737 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpzjp\" (UniqueName: \"kubernetes.io/projected/5c35f04b-688c-4f62-b55f-d757aeb29653-kube-api-access-gpzjp\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.769897 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.769808 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41d8bb2e-ae23-407a-9218-019bfc151181-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.769897 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.769864 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35f04b-688c-4f62-b55f-d757aeb29653-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.769987 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.769901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c35f04b-688c-4f62-b55f-d757aeb29653-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.769987 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.769929 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d8bb2e-ae23-407a-9218-019bfc151181-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.770067 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.770010 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d8bb2e-ae23-407a-9218-019bfc151181-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.770118 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.770081 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41d8bb2e-ae23-407a-9218-019bfc151181-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.770172 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.770113 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41d8bb2e-ae23-407a-9218-019bfc151181-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.770343 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.770327 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 09 06:37:33.770861 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.770844 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 09 06:37:33.771045 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.771027 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 09 06:37:33.774467 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.774451 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w892r\"" Sep 09 06:37:33.811176 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.811146 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw"] Sep 09 06:37:33.821608 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.821581 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h677h"] Sep 09 06:37:33.824930 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.824909 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.832115 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.832090 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 06:37:33.832891 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.832875 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 06:37:33.832953 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.832874 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jx2rm\"" Sep 09 06:37:33.833235 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.833221 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 06:37:33.871182 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871151 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-tls\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.871355 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871191 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35f04b-688c-4f62-b55f-d757aeb29653-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.871355 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871216 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rndt4\" (UniqueName: \"kubernetes.io/projected/daf1b317-03e2-4071-ae5e-abe256128da5-kube-api-access-rndt4\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.871355 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871243 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c35f04b-688c-4f62-b55f-d757aeb29653-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.871355 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d8bb2e-ae23-407a-9218-019bfc151181-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.871355 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871303 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d8bb2e-ae23-407a-9218-019bfc151181-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.871355 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:33.871317 2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 09 06:37:33.871355 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871330 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.871686 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871358 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41d8bb2e-ae23-407a-9218-019bfc151181-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.871686 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:33.871408 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c35f04b-688c-4f62-b55f-d757aeb29653-openshift-state-metrics-tls podName:5c35f04b-688c-4f62-b55f-d757aeb29653 nodeName:}" failed. No retries permitted until 2026-09-09 06:37:34.371386751 +0000 UTC m=+278.696197228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5c35f04b-688c-4f62-b55f-d757aeb29653-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-fb6wr" (UID: "5c35f04b-688c-4f62-b55f-d757aeb29653") : secret "openshift-state-metrics-tls" not found Sep 09 06:37:33.871686 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41d8bb2e-ae23-407a-9218-019bfc151181-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.871686 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871527 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/daf1b317-03e2-4071-ae5e-abe256128da5-root\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.871686 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871575 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c35f04b-688c-4f62-b55f-d757aeb29653-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.871686 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9vcf\" (UniqueName: \"kubernetes.io/projected/41d8bb2e-ae23-407a-9218-019bfc151181-kube-api-access-p9vcf\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.871686 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871624 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daf1b317-03e2-4071-ae5e-abe256128da5-sys\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.872045 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-textfile\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.872045 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpzjp\" (UniqueName: \"kubernetes.io/projected/5c35f04b-688c-4f62-b55f-d757aeb29653-kube-api-access-gpzjp\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.872045 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871826 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-accelerators-collector-config\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.872045 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871876 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daf1b317-03e2-4071-ae5e-abe256128da5-metrics-client-ca\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.872045 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41d8bb2e-ae23-407a-9218-019bfc151181-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.872045 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.871942 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-wtmp\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.872358 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.872060 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c35f04b-688c-4f62-b55f-d757aeb29653-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.872358 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.872120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41d8bb2e-ae23-407a-9218-019bfc151181-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.872469 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.872439 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41d8bb2e-ae23-407a-9218-019bfc151181-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.873159 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.873135 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41d8bb2e-ae23-407a-9218-019bfc151181-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.874815 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.874789 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c35f04b-688c-4f62-b55f-d757aeb29653-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.874929 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.874866 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d8bb2e-ae23-407a-9218-019bfc151181-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.874992 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.874953 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41d8bb2e-ae23-407a-9218-019bfc151181-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.890766 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.890744 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9vcf\" (UniqueName: \"kubernetes.io/projected/41d8bb2e-ae23-407a-9218-019bfc151181-kube-api-access-p9vcf\") pod \"kube-state-metrics-74cb57df7-6cgcw\" (UID: \"41d8bb2e-ae23-407a-9218-019bfc151181\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:33.899219 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.899193 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpzjp\" (UniqueName: \"kubernetes.io/projected/5c35f04b-688c-4f62-b55f-d757aeb29653-kube-api-access-gpzjp\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:33.972916 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.972880 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/daf1b317-03e2-4071-ae5e-abe256128da5-root\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973090 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.972922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daf1b317-03e2-4071-ae5e-abe256128da5-sys\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973090 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.972947 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-textfile\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973090 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973004 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/daf1b317-03e2-4071-ae5e-abe256128da5-root\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973090 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973007 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-accelerators-collector-config\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973090 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973021 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daf1b317-03e2-4071-ae5e-abe256128da5-sys\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973090 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daf1b317-03e2-4071-ae5e-abe256128da5-metrics-client-ca\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973090 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973091 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-wtmp\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973372 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973114 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-tls\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973372 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973151 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndt4\" (UniqueName: \"kubernetes.io/projected/daf1b317-03e2-4071-ae5e-abe256128da5-kube-api-access-rndt4\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973372 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973190 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973372 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973248 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-wtmp\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973372 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:33.973253 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 06:37:33.973372 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:33.973326 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-tls podName:daf1b317-03e2-4071-ae5e-abe256128da5 nodeName:}" failed. No retries permitted until 2026-09-09 06:37:34.473304545 +0000 UTC m=+278.798114995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-tls") pod "node-exporter-h677h" (UID: "daf1b317-03e2-4071-ae5e-abe256128da5") : secret "node-exporter-tls" not found Sep 09 06:37:33.973372 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973362 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-textfile\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973674 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973605 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-accelerators-collector-config\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.973707 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.973697 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daf1b317-03e2-4071-ae5e-abe256128da5-metrics-client-ca\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.975631 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.975612 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:33.987974 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:33.987909 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rndt4\" (UniqueName: \"kubernetes.io/projected/daf1b317-03e2-4071-ae5e-abe256128da5-kube-api-access-rndt4\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:34.076308 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.076264 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" Sep 09 06:37:34.373660 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.373639 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw"] Sep 09 06:37:34.375589 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:34.375568 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d8bb2e_ae23_407a_9218_019bfc151181.slice/crio-39a321e98f0e2109f5deeba057f809472bbbc648d058679b1bdff3bbf5fa876c WatchSource:0}: Error finding container 39a321e98f0e2109f5deeba057f809472bbbc648d058679b1bdff3bbf5fa876c: Status 404 returned error can't find the container with id 39a321e98f0e2109f5deeba057f809472bbbc648d058679b1bdff3bbf5fa876c Sep 09 06:37:34.376645 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.376627 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35f04b-688c-4f62-b55f-d757aeb29653-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:34.379280 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.379264 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35f04b-688c-4f62-b55f-d757aeb29653-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fb6wr\" (UID: \"5c35f04b-688c-4f62-b55f-d757aeb29653\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:34.477801 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.477770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-tls\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:34.480174 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.480146 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/daf1b317-03e2-4071-ae5e-abe256128da5-node-exporter-tls\") pod \"node-exporter-h677h\" (UID: \"daf1b317-03e2-4071-ae5e-abe256128da5\") " pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:34.643507 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.643429 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" Sep 09 06:37:34.733304 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.733276 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h677h" Sep 09 06:37:34.743937 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:34.743911 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf1b317_03e2_4071_ae5e_abe256128da5.slice/crio-87eb61c8e2805bba61c46bb185b2eb98d391f535c1663276824a9b736af9e76c WatchSource:0}: Error finding container 87eb61c8e2805bba61c46bb185b2eb98d391f535c1663276824a9b736af9e76c: Status 404 returned error can't find the container with id 87eb61c8e2805bba61c46bb185b2eb98d391f535c1663276824a9b736af9e76c Sep 09 06:37:34.950295 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.950258 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr"] Sep 09 06:37:34.953888 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:34.953858 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c35f04b_688c_4f62_b55f_d757aeb29653.slice/crio-0f8d0dabc3cc7d0c246cbfb2d42f569d12e003d77cb630ec737a086bca01868a WatchSource:0}: Error finding container 0f8d0dabc3cc7d0c246cbfb2d42f569d12e003d77cb630ec737a086bca01868a: Status 404 returned error can't find the container with id 0f8d0dabc3cc7d0c246cbfb2d42f569d12e003d77cb630ec737a086bca01868a Sep 09 06:37:34.994729 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.994704 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h677h" event={"ID":"daf1b317-03e2-4071-ae5e-abe256128da5","Type":"ContainerStarted","Data":"87eb61c8e2805bba61c46bb185b2eb98d391f535c1663276824a9b736af9e76c"} Sep 09 06:37:34.995990 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.995950 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" event={"ID":"41d8bb2e-ae23-407a-9218-019bfc151181","Type":"ContainerStarted","Data":"39a321e98f0e2109f5deeba057f809472bbbc648d058679b1bdff3bbf5fa876c"} Sep 09 06:37:34.997062 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:34.997040 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" event={"ID":"5c35f04b-688c-4f62-b55f-d757aeb29653","Type":"ContainerStarted","Data":"0f8d0dabc3cc7d0c246cbfb2d42f569d12e003d77cb630ec737a086bca01868a"} Sep 09 06:37:36.002865 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.002825 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" event={"ID":"5c35f04b-688c-4f62-b55f-d757aeb29653","Type":"ContainerStarted","Data":"87ed00fc210142c7717954e9d2370e66e617ef15db3f09df0e34a4ebb615bacc"} Sep 09 06:37:36.003309 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.002905 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" event={"ID":"5c35f04b-688c-4f62-b55f-d757aeb29653","Type":"ContainerStarted","Data":"ee445403fb5b21c557c95b98319c0db3d36bcbf7152c793aa018a8b95bd37435"} Sep 09 06:37:36.052604 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.052566 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-7nh4z"] Sep 09 06:37:36.055924 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.055903 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-7nh4z" Sep 09 06:37:36.058853 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.058811 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 06:37:36.059253 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.059233 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 06:37:36.059575 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.059556 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bk2xr\"" Sep 09 06:37:36.081618 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.081592 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-7nh4z"] Sep 09 06:37:36.090086 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.090063 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq9qj\" (UniqueName: \"kubernetes.io/projected/dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919-kube-api-access-hq9qj\") pod \"downloads-85d98c998f-7nh4z\" (UID: \"dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919\") " pod="openshift-console/downloads-85d98c998f-7nh4z" Sep 09 06:37:36.191017 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.190941 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq9qj\" (UniqueName: \"kubernetes.io/projected/dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919-kube-api-access-hq9qj\") pod \"downloads-85d98c998f-7nh4z\" (UID: \"dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919\") " pod="openshift-console/downloads-85d98c998f-7nh4z" Sep 09 06:37:36.204121 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.204091 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq9qj\" (UniqueName: \"kubernetes.io/projected/dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919-kube-api-access-hq9qj\") pod \"downloads-85d98c998f-7nh4z\" (UID: \"dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919\") " pod="openshift-console/downloads-85d98c998f-7nh4z" Sep 09 06:37:36.367097 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.367061 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-7nh4z" Sep 09 06:37:36.635707 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:36.635589 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d8257" podUID="784894d0-7e1e-4611-9e85-c737ddbe3cad" Sep 09 06:37:36.635707 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:37:36.635600 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" podUID="2500df92-212c-4be0-8813-972d17dc1952" Sep 09 06:37:36.707465 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:36.707215 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc238f64_8c8b_4a1f_a9fa_d3a3c6ab2919.slice/crio-051af993cef9904e44a94439aea17c8f8d9bd067bee9c6247bff681341631740 WatchSource:0}: Error finding container 051af993cef9904e44a94439aea17c8f8d9bd067bee9c6247bff681341631740: Status 404 returned error can't find the container with id 051af993cef9904e44a94439aea17c8f8d9bd067bee9c6247bff681341631740 Sep 09 06:37:36.707631 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:36.707612 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-7nh4z"] Sep 09 06:37:37.009259 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.009163 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" event={"ID":"5c35f04b-688c-4f62-b55f-d757aeb29653","Type":"ContainerStarted","Data":"869c462e9756287b13d590c08a7d30f3ab03fdb4cf575e87956f8726c5610b0e"} Sep 09 06:37:37.010744 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.010719 2563 generic.go:356] "Generic (PLEG): container finished" podID="daf1b317-03e2-4071-ae5e-abe256128da5" containerID="c7ebdde545bb6cec97ff4b7657bf293f2a8f86c7e21f5c87870d7a8fb3417f89" exitCode=0 Sep 09 06:37:37.010853 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.010779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h677h" event={"ID":"daf1b317-03e2-4071-ae5e-abe256128da5","Type":"ContainerDied","Data":"c7ebdde545bb6cec97ff4b7657bf293f2a8f86c7e21f5c87870d7a8fb3417f89"} Sep 09 06:37:37.013068 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.013047 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" event={"ID":"41d8bb2e-ae23-407a-9218-019bfc151181","Type":"ContainerStarted","Data":"1395a3224e6eb8ba57b0ec51780fdd04e31c54c195e26e91e954e33ea3d78dfa"} Sep 09 06:37:37.013176 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.013098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" event={"ID":"41d8bb2e-ae23-407a-9218-019bfc151181","Type":"ContainerStarted","Data":"8fd4097854272db38e7c7fc6f7e6af43edc7350e32d58143478ee82b2dae1d44"} Sep 09 06:37:37.013176 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.013125 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" event={"ID":"41d8bb2e-ae23-407a-9218-019bfc151181","Type":"ContainerStarted","Data":"34c9cf5c3772395a225b30bce67e8138aa31b8eb986a7c1f88a72876d1de7766"} Sep 09 06:37:37.014385 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.014364 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-7nh4z" event={"ID":"dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919","Type":"ContainerStarted","Data":"051af993cef9904e44a94439aea17c8f8d9bd067bee9c6247bff681341631740"} Sep 09 06:37:37.014385 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.014384 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:37:37.014548 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.014518 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d8257" Sep 09 06:37:37.037206 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.037161 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fb6wr" podStartSLOduration=2.693633378 podStartE2EDuration="4.037146277s" totalImagesPullingTime="1.343512899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:37.035902754 +0000 UTC m=+281.360713237" watchObservedRunningTime="2026-09-09 06:37:37.037146277 +0000 UTC m=+281.361956733" Sep 09 06:37:37.099932 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:37.099872 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6cgcw" podStartSLOduration=2.060938589 podStartE2EDuration="4.099850168s" totalImagesPullingTime="2.038911579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:37.09846314 +0000 UTC m=+281.423273598" watchObservedRunningTime="2026-09-09 06:37:37.099850168 +0000 UTC m=+281.424660631" Sep 09 06:37:38.021467 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.021434 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h677h" event={"ID":"daf1b317-03e2-4071-ae5e-abe256128da5","Type":"ContainerStarted","Data":"0cf06f5a7f009ef4d1c04ea1ab5eec73438c70765d95b1d665023062e788a95e"} Sep 09 06:37:38.021919 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.021487 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h677h" event={"ID":"daf1b317-03e2-4071-ae5e-abe256128da5","Type":"ContainerStarted","Data":"d8868565570fe55a3b9409accfda5c3de18aae73e5b5399c7cdbe1d59796781d"} Sep 09 06:37:38.044761 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.044712 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h677h" podStartSLOduration=3.296569168 podStartE2EDuration="5.044692572s" totalImagesPullingTime="1.662254182s" totalInitContainerRuntime="85.869222ms" isStatefulPod=true podCreationTimestamp="2026-09-09 06:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:38.043493794 +0000 UTC m=+282.368304274" watchObservedRunningTime="2026-09-09 06:37:38.044692572 +0000 UTC m=+282.369503029" Sep 09 06:37:38.318765 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.318687 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-75f55d8478-dh8kl"] Sep 09 06:37:38.325412 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.325389 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.330027 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.329880 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 06:37:38.330027 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.329881 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-m5jhn\"" Sep 09 06:37:38.330027 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.329915 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 09 06:37:38.330027 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.329933 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cn0rjo2unkqco\"" Sep 09 06:37:38.330027 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.330027 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 09 06:37:38.330405 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.329891 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 09 06:37:38.337826 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.337807 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75f55d8478-dh8kl"] Sep 09 06:37:38.412427 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.412396 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b89c9acd-a35c-4085-af18-5a7d4b183d45-secret-metrics-server-tls\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.412627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.412470 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b89c9acd-a35c-4085-af18-5a7d4b183d45-audit-log\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.412627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.412494 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhkf\" (UniqueName: \"kubernetes.io/projected/b89c9acd-a35c-4085-af18-5a7d4b183d45-kube-api-access-hkhkf\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.412627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.412522 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b89c9acd-a35c-4085-af18-5a7d4b183d45-secret-metrics-server-client-certs\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.412627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.412584 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b89c9acd-a35c-4085-af18-5a7d4b183d45-metrics-server-audit-profiles\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.412627 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.412613 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b89c9acd-a35c-4085-af18-5a7d4b183d45-client-ca-bundle\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.412848 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.412682 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b89c9acd-a35c-4085-af18-5a7d4b183d45-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.418769 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.418742 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m"] Sep 09 06:37:38.423025 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.423009 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:38.426872 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.426854 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jfzk7\"" Sep 09 06:37:38.426996 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.426948 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 09 06:37:38.434661 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.434638 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m"] Sep 09 06:37:38.513847 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.513811 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b89c9acd-a35c-4085-af18-5a7d4b183d45-audit-log\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.514011 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.513853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhkf\" (UniqueName: \"kubernetes.io/projected/b89c9acd-a35c-4085-af18-5a7d4b183d45-kube-api-access-hkhkf\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.514011 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.513883 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8hr\" (UniqueName: \"kubernetes.io/projected/5baef439-4b19-4422-acce-8b0ca7bade33-kube-api-access-4g8hr\") pod \"monitoring-plugin-85bbbbbf4-pvt4m\" (UID: \"5baef439-4b19-4422-acce-8b0ca7bade33\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:38.514011 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.513921 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b89c9acd-a35c-4085-af18-5a7d4b183d45-secret-metrics-server-client-certs\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.514011 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.513993 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b89c9acd-a35c-4085-af18-5a7d4b183d45-metrics-server-audit-profiles\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.514281 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.514021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b89c9acd-a35c-4085-af18-5a7d4b183d45-client-ca-bundle\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.514281 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.514057 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b89c9acd-a35c-4085-af18-5a7d4b183d45-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.514281 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.514091 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b89c9acd-a35c-4085-af18-5a7d4b183d45-secret-metrics-server-tls\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.514281 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.514148 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5baef439-4b19-4422-acce-8b0ca7bade33-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-pvt4m\" (UID: \"5baef439-4b19-4422-acce-8b0ca7bade33\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:38.514281 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.514256 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b89c9acd-a35c-4085-af18-5a7d4b183d45-audit-log\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.514985 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.514939 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b89c9acd-a35c-4085-af18-5a7d4b183d45-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.515157 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.515089 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b89c9acd-a35c-4085-af18-5a7d4b183d45-metrics-server-audit-profiles\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.517091 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.517069 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b89c9acd-a35c-4085-af18-5a7d4b183d45-secret-metrics-server-client-certs\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.517199 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.517180 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b89c9acd-a35c-4085-af18-5a7d4b183d45-client-ca-bundle\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.517257 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.517180 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b89c9acd-a35c-4085-af18-5a7d4b183d45-secret-metrics-server-tls\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.527849 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.527825 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhkf\" (UniqueName: \"kubernetes.io/projected/b89c9acd-a35c-4085-af18-5a7d4b183d45-kube-api-access-hkhkf\") pod \"metrics-server-75f55d8478-dh8kl\" (UID: \"b89c9acd-a35c-4085-af18-5a7d4b183d45\") " pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.615311 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.615273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5baef439-4b19-4422-acce-8b0ca7bade33-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-pvt4m\" (UID: \"5baef439-4b19-4422-acce-8b0ca7bade33\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:38.615496 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.615335 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8hr\" (UniqueName: \"kubernetes.io/projected/5baef439-4b19-4422-acce-8b0ca7bade33-kube-api-access-4g8hr\") pod \"monitoring-plugin-85bbbbbf4-pvt4m\" (UID: \"5baef439-4b19-4422-acce-8b0ca7bade33\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:38.618443 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.618421 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5baef439-4b19-4422-acce-8b0ca7bade33-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-pvt4m\" (UID: \"5baef439-4b19-4422-acce-8b0ca7bade33\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:38.631787 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.631767 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8hr\" (UniqueName: \"kubernetes.io/projected/5baef439-4b19-4422-acce-8b0ca7bade33-kube-api-access-4g8hr\") pod \"monitoring-plugin-85bbbbbf4-pvt4m\" (UID: \"5baef439-4b19-4422-acce-8b0ca7bade33\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:38.636578 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.636554 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:38.733692 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.733661 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:38.919825 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:38.919789 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75f55d8478-dh8kl"] Sep 09 06:37:38.922305 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:38.922274 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89c9acd_a35c_4085_af18_5a7d4b183d45.slice/crio-202c85d2e5a74ef26c2ca246b71effa1a19ad5ad5c9551e49abe7bf8b9131316 WatchSource:0}: Error finding container 202c85d2e5a74ef26c2ca246b71effa1a19ad5ad5c9551e49abe7bf8b9131316: Status 404 returned error can't find the container with id 202c85d2e5a74ef26c2ca246b71effa1a19ad5ad5c9551e49abe7bf8b9131316 Sep 09 06:37:39.015994 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:39.015921 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m"] Sep 09 06:37:39.018149 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:39.018110 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baef439_4b19_4422_acce_8b0ca7bade33.slice/crio-e7d32b7fdbe7639d37bbac041e453167b51170a75710ca40fd70afea737ed1f3 WatchSource:0}: Error finding container e7d32b7fdbe7639d37bbac041e453167b51170a75710ca40fd70afea737ed1f3: Status 404 returned error can't find the container with id e7d32b7fdbe7639d37bbac041e453167b51170a75710ca40fd70afea737ed1f3 Sep 09 06:37:39.029325 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:39.029297 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" event={"ID":"b89c9acd-a35c-4085-af18-5a7d4b183d45","Type":"ContainerStarted","Data":"202c85d2e5a74ef26c2ca246b71effa1a19ad5ad5c9551e49abe7bf8b9131316"} Sep 09 06:37:39.030588 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:39.030555 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" event={"ID":"5baef439-4b19-4422-acce-8b0ca7bade33","Type":"ContainerStarted","Data":"e7d32b7fdbe7639d37bbac041e453167b51170a75710ca40fd70afea737ed1f3"} Sep 09 06:37:40.532953 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:40.532913 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:37:40.533431 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:40.533012 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:37:40.538095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:40.538033 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/784894d0-7e1e-4611-9e85-c737ddbe3cad-metrics-tls\") pod \"dns-default-d8257\" (UID: \"784894d0-7e1e-4611-9e85-c737ddbe3cad\") " pod="openshift-dns/dns-default-d8257" Sep 09 06:37:40.538876 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:40.538833 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"image-registry-69496f8585-qwx6b\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:37:40.618748 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:40.618712 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgrss\"" Sep 09 06:37:40.618927 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:40.618893 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8bmbn\"" Sep 09 06:37:40.626341 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:40.625242 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:37:40.626341 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:40.625499 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d8257" Sep 09 06:37:41.469940 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:41.469868 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d8257"] Sep 09 06:37:41.474881 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:41.474795 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784894d0_7e1e_4611_9e85_c737ddbe3cad.slice/crio-d0d9d65c50a1919f10ba2888754dbb816c858c2ef1f61f4f9e92c0e149a000e6 WatchSource:0}: Error finding container d0d9d65c50a1919f10ba2888754dbb816c858c2ef1f61f4f9e92c0e149a000e6: Status 404 returned error can't find the container with id d0d9d65c50a1919f10ba2888754dbb816c858c2ef1f61f4f9e92c0e149a000e6 Sep 09 06:37:41.476299 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:41.476219 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69496f8585-qwx6b"] Sep 09 06:37:41.478898 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:41.478875 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2500df92_212c_4be0_8813_972d17dc1952.slice/crio-b1c49f0c8ddc21fde753440a075c4ca529a3a2ba4988b4968511ee0cf22a4571 WatchSource:0}: Error finding container b1c49f0c8ddc21fde753440a075c4ca529a3a2ba4988b4968511ee0cf22a4571: Status 404 returned error can't find the container with id b1c49f0c8ddc21fde753440a075c4ca529a3a2ba4988b4968511ee0cf22a4571 Sep 09 06:37:42.052354 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.052318 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" event={"ID":"5baef439-4b19-4422-acce-8b0ca7bade33","Type":"ContainerStarted","Data":"9ae1de14ced66c07daa227a18993b8d02179ae20bdc77cdcc3baf7c6d1dc4173"} Sep 09 06:37:42.052710 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.052683 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:42.055147 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.055121 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" event={"ID":"2500df92-212c-4be0-8813-972d17dc1952","Type":"ContainerStarted","Data":"aeea2475e937173f44168737e23f1522265144058ca75925667180102a50ca21"} Sep 09 06:37:42.055270 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.055168 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" event={"ID":"2500df92-212c-4be0-8813-972d17dc1952","Type":"ContainerStarted","Data":"b1c49f0c8ddc21fde753440a075c4ca529a3a2ba4988b4968511ee0cf22a4571"} Sep 09 06:37:42.055347 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.055330 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:37:42.058089 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.058024 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d8257" event={"ID":"784894d0-7e1e-4611-9e85-c737ddbe3cad","Type":"ContainerStarted","Data":"d0d9d65c50a1919f10ba2888754dbb816c858c2ef1f61f4f9e92c0e149a000e6"} Sep 09 06:37:42.059739 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.059721 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" Sep 09 06:37:42.061665 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.061641 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" event={"ID":"b89c9acd-a35c-4085-af18-5a7d4b183d45","Type":"ContainerStarted","Data":"d2474b7cdc8c721c92ccb4591ea9d905af93d9720a8573c91685d4c047848e4f"} Sep 09 06:37:42.071615 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.071561 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pvt4m" podStartSLOduration=1.907143675 podStartE2EDuration="4.071545789s" totalImagesPullingTime="2.164402114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:42.069908373 +0000 UTC m=+286.394718830" watchObservedRunningTime="2026-09-09 06:37:42.071545789 +0000 UTC m=+286.396356244" Sep 09 06:37:42.104350 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.103415 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" podStartSLOduration=1.844977431 podStartE2EDuration="4.103401756s" totalImagesPullingTime="2.258424325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:42.099805318 +0000 UTC m=+286.424615774" watchObservedRunningTime="2026-09-09 06:37:42.103401756 +0000 UTC m=+286.428212217" Sep 09 06:37:42.152728 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:42.152673 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" podStartSLOduration=286.152651649 podStartE2EDuration="4m46.152651649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:42.150844733 +0000 UTC m=+286.475655234" watchObservedRunningTime="2026-09-09 06:37:42.152651649 +0000 UTC m=+286.477462106" Sep 09 06:37:44.073765 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:44.073722 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d8257" event={"ID":"784894d0-7e1e-4611-9e85-c737ddbe3cad","Type":"ContainerStarted","Data":"9456213ffe83172153da3f95b294e4cea433f7641457c02964de4dde0811c11b"} Sep 09 06:37:44.074245 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:44.073786 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d8257" event={"ID":"784894d0-7e1e-4611-9e85-c737ddbe3cad","Type":"ContainerStarted","Data":"5cba816c8bba49caf28153fcff69ca6f00a19cae0d7bb08239605cce46acb28e"} Sep 09 06:37:44.098719 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:44.098660 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d8257" podStartSLOduration=252.275623196 podStartE2EDuration="4m14.09864034s" totalImagesPullingTime="1.823017144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:44.097117925 +0000 UTC m=+288.421928390" watchObservedRunningTime="2026-09-09 06:37:44.09864034 +0000 UTC m=+288.423450798" Sep 09 06:37:45.083232 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:45.080254 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d8257" Sep 09 06:37:47.760140 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.760098 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-564f69d889-5kvw4"] Sep 09 06:37:47.763411 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.763387 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.766608 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.766573 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 06:37:47.766703 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.766687 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 06:37:47.766789 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.766774 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 06:37:47.767747 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.767725 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-72f7v\"" Sep 09 06:37:47.767826 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.767776 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 06:37:47.767826 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.767776 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 06:37:47.780104 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.780078 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564f69d889-5kvw4"] Sep 09 06:37:47.896796 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.896763 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tj6j\" (UniqueName: \"kubernetes.io/projected/0458fec3-51a2-44eb-9f26-8c5c3b663d74-kube-api-access-5tj6j\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.896994 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.896802 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-config\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.896994 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.896860 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-oauth-serving-cert\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.896994 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.896884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-serving-cert\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.897162 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.897002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-service-ca\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.897162 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.897041 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-oauth-config\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.998449 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.998409 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-oauth-config\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.998612 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.998472 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tj6j\" (UniqueName: \"kubernetes.io/projected/0458fec3-51a2-44eb-9f26-8c5c3b663d74-kube-api-access-5tj6j\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.998612 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.998495 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-config\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.998612 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.998533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-oauth-serving-cert\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.998612 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.998558 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-serving-cert\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.998830 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.998622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-service-ca\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.999458 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.999423 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-config\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.999575 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.999457 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-service-ca\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:47.999575 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:47.999473 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-oauth-serving-cert\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:48.001578 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:48.001556 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-serving-cert\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:48.001759 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:48.001740 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-oauth-config\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:48.011223 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:48.011162 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tj6j\" (UniqueName: \"kubernetes.io/projected/0458fec3-51a2-44eb-9f26-8c5c3b663d74-kube-api-access-5tj6j\") pod \"console-564f69d889-5kvw4\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:48.074674 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:48.074648 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:37:54.118955 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:54.118924 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-7nh4z" event={"ID":"dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919","Type":"ContainerStarted","Data":"af3a8dde1719edffdde3c67c271d1bc9c527dffa91188d2915238a6763db8032"} Sep 09 06:37:54.119294 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:54.119095 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-7nh4z" Sep 09 06:37:54.120511 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:54.120480 2563 patch_prober.go:28] interesting pod/downloads-85d98c998f-7nh4z container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" start-of-body= Sep 09 06:37:54.120590 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:54.120532 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-7nh4z" podUID="dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" Sep 09 06:37:54.170734 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:54.170423 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-7nh4z" podStartSLOduration=0.88500139 podStartE2EDuration="18.170402174s" totalImagesPullingTime="17.285400784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:54.145755715 +0000 UTC m=+298.470566171" watchObservedRunningTime="2026-09-09 06:37:54.170402174 +0000 UTC m=+298.495212633" Sep 09 06:37:54.171387 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:54.171369 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564f69d889-5kvw4"] Sep 09 06:37:54.174311 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:54.174264 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0458fec3_51a2_44eb_9f26_8c5c3b663d74.slice/crio-c916fc2aca908441ac18af89bec5c217ddc8ba3d12e71e9cbe296ee492e8eba0 WatchSource:0}: Error finding container c916fc2aca908441ac18af89bec5c217ddc8ba3d12e71e9cbe296ee492e8eba0: Status 404 returned error can't find the container with id c916fc2aca908441ac18af89bec5c217ddc8ba3d12e71e9cbe296ee492e8eba0 Sep 09 06:37:55.087749 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:55.087719 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d8257" Sep 09 06:37:55.127259 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:55.127223 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564f69d889-5kvw4" event={"ID":"0458fec3-51a2-44eb-9f26-8c5c3b663d74","Type":"ContainerStarted","Data":"c916fc2aca908441ac18af89bec5c217ddc8ba3d12e71e9cbe296ee492e8eba0"} Sep 09 06:37:55.136253 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:55.136227 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-7nh4z" Sep 09 06:37:56.150756 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:56.150618 2563 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 06:37:57.170754 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.138106 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d6ff4c55-4mpfr"] Sep 09 06:37:57.170754 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.157714 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d6ff4c55-4mpfr"] Sep 09 06:37:57.170754 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.157828 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.170754 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.168131 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 06:37:57.285321 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.285287 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshdd\" (UniqueName: \"kubernetes.io/projected/ff526f80-fc41-4506-9938-e54a8a7f265c-kube-api-access-rshdd\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.285525 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.285351 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-service-ca\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.285525 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.285380 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-serving-cert\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.285525 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.285444 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-console-config\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.285525 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.285463 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-oauth-serving-cert\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.285525 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.285489 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-oauth-config\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.285525 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.285514 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-trusted-ca-bundle\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.387435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.386845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-trusted-ca-bundle\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.387435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.386916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshdd\" (UniqueName: \"kubernetes.io/projected/ff526f80-fc41-4506-9938-e54a8a7f265c-kube-api-access-rshdd\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.387435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.386994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-service-ca\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.387435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.387181 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-serving-cert\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.387435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.387262 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-console-config\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.387435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.387291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-oauth-serving-cert\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.387435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.387313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-oauth-config\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.387920 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.387897 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-trusted-ca-bundle\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.388007 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.387903 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-console-config\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.388301 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.388261 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-oauth-serving-cert\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.388564 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.388541 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-service-ca\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.390901 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.390881 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-oauth-config\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.391087 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.390903 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-serving-cert\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.397667 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.397624 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshdd\" (UniqueName: \"kubernetes.io/projected/ff526f80-fc41-4506-9938-e54a8a7f265c-kube-api-access-rshdd\") pod \"console-57d6ff4c55-4mpfr\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:57.475854 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:57.475727 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:37:58.390288 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:58.390139 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d6ff4c55-4mpfr"] Sep 09 06:37:58.393187 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:37:58.393159 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff526f80_fc41_4506_9938_e54a8a7f265c.slice/crio-f1b0fa7f07512bc96aa6ce4225e1afe331c34b4b09added239eea435cf1b6a82 WatchSource:0}: Error finding container f1b0fa7f07512bc96aa6ce4225e1afe331c34b4b09added239eea435cf1b6a82: Status 404 returned error can't find the container with id f1b0fa7f07512bc96aa6ce4225e1afe331c34b4b09added239eea435cf1b6a82 Sep 09 06:37:58.637117 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:58.637074 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:58.637308 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:58.637158 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:37:59.151356 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:59.151310 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d6ff4c55-4mpfr" event={"ID":"ff526f80-fc41-4506-9938-e54a8a7f265c","Type":"ContainerStarted","Data":"63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c"} Sep 09 06:37:59.151547 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:59.151384 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d6ff4c55-4mpfr" event={"ID":"ff526f80-fc41-4506-9938-e54a8a7f265c","Type":"ContainerStarted","Data":"f1b0fa7f07512bc96aa6ce4225e1afe331c34b4b09added239eea435cf1b6a82"} Sep 09 06:37:59.153226 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:59.153198 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564f69d889-5kvw4" event={"ID":"0458fec3-51a2-44eb-9f26-8c5c3b663d74","Type":"ContainerStarted","Data":"d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2"} Sep 09 06:37:59.172955 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:59.172903 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57d6ff4c55-4mpfr" podStartSLOduration=2.172882964 podStartE2EDuration="2.172882964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:59.170717352 +0000 UTC m=+303.495527851" watchObservedRunningTime="2026-09-09 06:37:59.172882964 +0000 UTC m=+303.497693421" Sep 09 06:37:59.189247 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:37:59.189197 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-564f69d889-5kvw4" podStartSLOduration=7.960369565 podStartE2EDuration="12.18918236s" totalImagesPullingTime="4.228812795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:37:59.187846064 +0000 UTC m=+303.512656534" watchObservedRunningTime="2026-09-09 06:37:59.18918236 +0000 UTC m=+303.513992841" Sep 09 06:38:00.630733 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:00.630697 2563 patch_prober.go:28] interesting pod/image-registry-69496f8585-qwx6b container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 06:38:00.631215 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:00.630749 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" podUID="2500df92-212c-4be0-8813-972d17dc1952" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 06:38:03.071352 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:03.071320 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:38:04.176598 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:04.176566 2563 generic.go:356] "Generic (PLEG): container finished" podID="ace615ac-3fae-4d50-9e0b-e5b3d149f51f" containerID="fe170b8612d0818ac9b126af30fbc4604d5208a54f3e784d4b875791be7fa743" exitCode=0 Sep 09 06:38:04.177014 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:04.176638 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" event={"ID":"ace615ac-3fae-4d50-9e0b-e5b3d149f51f","Type":"ContainerDied","Data":"fe170b8612d0818ac9b126af30fbc4604d5208a54f3e784d4b875791be7fa743"} Sep 09 06:38:04.177014 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:04.176904 2563 scope.go:117] "RemoveContainer" containerID="fe170b8612d0818ac9b126af30fbc4604d5208a54f3e784d4b875791be7fa743" Sep 09 06:38:05.182148 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:05.182118 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4h66" event={"ID":"ace615ac-3fae-4d50-9e0b-e5b3d149f51f","Type":"ContainerStarted","Data":"ce4c0e3380f19dcd1a239971632edc994608b17495b3013dd00dde31d1b17b14"} Sep 09 06:38:07.477243 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:07.476792 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:38:07.477875 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:07.477847 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:38:07.481683 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:07.481656 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:38:07.538792 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:07.538758 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69496f8585-qwx6b"] Sep 09 06:38:08.075487 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:08.075453 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:38:08.075682 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:08.075521 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:38:08.080048 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:08.080025 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:38:08.197147 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:08.197122 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:38:08.197554 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:08.197538 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:38:08.277080 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:08.277051 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564f69d889-5kvw4"] Sep 09 06:38:18.642275 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:18.642246 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:38:18.646024 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:18.646003 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-75f55d8478-dh8kl" Sep 09 06:38:19.835737 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:19.835707 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8t8b_02718a56-6748-4bb7-9cef-01d5fe02132a/serve-healthcheck-canary/0.log" Sep 09 06:38:20.229506 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:20.229472 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-hlb8s_77cf1306-5823-4ed6-826c-2fad59e20080/cluster-samples-operator/0.log" Sep 09 06:38:20.427171 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:20.427136 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-hlb8s_77cf1306-5823-4ed6-826c-2fad59e20080/cluster-samples-operator-watch/0.log" Sep 09 06:38:24.257402 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:24.257374 2563 generic.go:356] "Generic (PLEG): container finished" podID="af365614-5695-450a-a777-f25e24f4d564" containerID="96a193646bc17d0ea8a4681fc9daca41c2e1202916061c64f6832dc9966060de" exitCode=0 Sep 09 06:38:24.257835 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:24.257448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" event={"ID":"af365614-5695-450a-a777-f25e24f4d564","Type":"ContainerDied","Data":"96a193646bc17d0ea8a4681fc9daca41c2e1202916061c64f6832dc9966060de"} Sep 09 06:38:24.257835 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:24.257730 2563 scope.go:117] "RemoveContainer" containerID="96a193646bc17d0ea8a4681fc9daca41c2e1202916061c64f6832dc9966060de" Sep 09 06:38:25.263427 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:25.263393 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-knzv9" event={"ID":"af365614-5695-450a-a777-f25e24f4d564","Type":"ContainerStarted","Data":"013f23b71635a8468b6a95ab426677722cab3bfb2dad4ff50afefac8f187ee0f"} Sep 09 06:38:32.565143 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:32.565102 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" podUID="2500df92-212c-4be0-8813-972d17dc1952" containerName="registry" containerID="cri-o://aeea2475e937173f44168737e23f1522265144058ca75925667180102a50ca21" gracePeriod=30 Sep 09 06:38:33.067932 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:33.067897 2563 patch_prober.go:28] interesting pod/image-registry-69496f8585-qwx6b container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" start-of-body= Sep 09 06:38:33.068111 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:33.067950 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" podUID="2500df92-212c-4be0-8813-972d17dc1952" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" Sep 09 06:38:34.305824 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.305785 2563 generic.go:356] "Generic (PLEG): container finished" podID="2500df92-212c-4be0-8813-972d17dc1952" containerID="aeea2475e937173f44168737e23f1522265144058ca75925667180102a50ca21" exitCode=0 Sep 09 06:38:34.306245 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.305848 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" event={"ID":"2500df92-212c-4be0-8813-972d17dc1952","Type":"ContainerDied","Data":"aeea2475e937173f44168737e23f1522265144058ca75925667180102a50ca21"} Sep 09 06:38:34.367721 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.367698 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:38:34.388023 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.387998 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-bound-sa-token\") pod \"2500df92-212c-4be0-8813-972d17dc1952\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " Sep 09 06:38:34.388138 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.388034 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") pod \"2500df92-212c-4be0-8813-972d17dc1952\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " Sep 09 06:38:34.388138 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.388059 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52h9p\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-kube-api-access-52h9p\") pod \"2500df92-212c-4be0-8813-972d17dc1952\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " Sep 09 06:38:34.388138 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.388086 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-installation-pull-secrets\") pod \"2500df92-212c-4be0-8813-972d17dc1952\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " Sep 09 06:38:34.388138 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.388116 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2500df92-212c-4be0-8813-972d17dc1952-ca-trust-extracted\") pod \"2500df92-212c-4be0-8813-972d17dc1952\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " Sep 09 06:38:34.388330 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.388136 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-image-registry-private-configuration\") pod \"2500df92-212c-4be0-8813-972d17dc1952\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " Sep 09 06:38:34.388330 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.388163 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-trusted-ca\") pod \"2500df92-212c-4be0-8813-972d17dc1952\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " Sep 09 06:38:34.388330 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.388185 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-registry-certificates\") pod \"2500df92-212c-4be0-8813-972d17dc1952\" (UID: \"2500df92-212c-4be0-8813-972d17dc1952\") " Sep 09 06:38:34.388820 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.388753 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2500df92-212c-4be0-8813-972d17dc1952" (UID: "2500df92-212c-4be0-8813-972d17dc1952"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:38:34.388820 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.388769 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2500df92-212c-4be0-8813-972d17dc1952" (UID: "2500df92-212c-4be0-8813-972d17dc1952"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:38:34.391164 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.391099 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2500df92-212c-4be0-8813-972d17dc1952" (UID: "2500df92-212c-4be0-8813-972d17dc1952"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:38:34.392500 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.392165 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2500df92-212c-4be0-8813-972d17dc1952" (UID: "2500df92-212c-4be0-8813-972d17dc1952"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:38:34.392500 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.392424 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-kube-api-access-52h9p" (OuterVolumeSpecName: "kube-api-access-52h9p") pod "2500df92-212c-4be0-8813-972d17dc1952" (UID: "2500df92-212c-4be0-8813-972d17dc1952"). InnerVolumeSpecName "kube-api-access-52h9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:38:34.392500 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.392164 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2500df92-212c-4be0-8813-972d17dc1952" (UID: "2500df92-212c-4be0-8813-972d17dc1952"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:38:34.392863 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.392835 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2500df92-212c-4be0-8813-972d17dc1952" (UID: "2500df92-212c-4be0-8813-972d17dc1952"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:38:34.400712 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.400674 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2500df92-212c-4be0-8813-972d17dc1952-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2500df92-212c-4be0-8813-972d17dc1952" (UID: "2500df92-212c-4be0-8813-972d17dc1952"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:38:34.489096 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.489017 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-bound-sa-token\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:34.489096 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.489042 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:34.489096 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.489050 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52h9p\" (UniqueName: \"kubernetes.io/projected/2500df92-212c-4be0-8813-972d17dc1952-kube-api-access-52h9p\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:34.489096 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.489059 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-installation-pull-secrets\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:34.489096 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.489067 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2500df92-212c-4be0-8813-972d17dc1952-ca-trust-extracted\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:34.489096 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.489076 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2500df92-212c-4be0-8813-972d17dc1952-image-registry-private-configuration\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:34.489096 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.489084 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-trusted-ca\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:34.489408 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:34.489115 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2500df92-212c-4be0-8813-972d17dc1952-registry-certificates\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:35.221010 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.220952 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-564f69d889-5kvw4" podUID="0458fec3-51a2-44eb-9f26-8c5c3b663d74" containerName="console" containerID="cri-o://d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2" gracePeriod=15 Sep 09 06:38:35.311216 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.311184 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" event={"ID":"2500df92-212c-4be0-8813-972d17dc1952","Type":"ContainerDied","Data":"b1c49f0c8ddc21fde753440a075c4ca529a3a2ba4988b4968511ee0cf22a4571"} Sep 09 06:38:35.311612 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.311239 2563 scope.go:117] "RemoveContainer" containerID="aeea2475e937173f44168737e23f1522265144058ca75925667180102a50ca21" Sep 09 06:38:35.311612 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.311186 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69496f8585-qwx6b" Sep 09 06:38:35.334577 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.334554 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69496f8585-qwx6b"] Sep 09 06:38:35.338880 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.338860 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69496f8585-qwx6b"] Sep 09 06:38:35.506508 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.506481 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:38:35.598606 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.598572 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tj6j\" (UniqueName: \"kubernetes.io/projected/0458fec3-51a2-44eb-9f26-8c5c3b663d74-kube-api-access-5tj6j\") pod \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " Sep 09 06:38:35.598763 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.598622 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-oauth-serving-cert\") pod \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " Sep 09 06:38:35.598763 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.598660 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-serving-cert\") pod \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " Sep 09 06:38:35.598763 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.598680 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-service-ca\") pod \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " Sep 09 06:38:35.598763 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.598697 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-oauth-config\") pod \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " Sep 09 06:38:35.598763 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.598716 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-config\") pod \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\" (UID: \"0458fec3-51a2-44eb-9f26-8c5c3b663d74\") " Sep 09 06:38:35.599125 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.599093 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-service-ca" (OuterVolumeSpecName: "service-ca") pod "0458fec3-51a2-44eb-9f26-8c5c3b663d74" (UID: "0458fec3-51a2-44eb-9f26-8c5c3b663d74"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:38:35.599197 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.599117 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0458fec3-51a2-44eb-9f26-8c5c3b663d74" (UID: "0458fec3-51a2-44eb-9f26-8c5c3b663d74"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:38:35.599197 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.599164 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-config" (OuterVolumeSpecName: "console-config") pod "0458fec3-51a2-44eb-9f26-8c5c3b663d74" (UID: "0458fec3-51a2-44eb-9f26-8c5c3b663d74"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:38:35.601108 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.601084 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0458fec3-51a2-44eb-9f26-8c5c3b663d74" (UID: "0458fec3-51a2-44eb-9f26-8c5c3b663d74"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:38:35.601209 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.601127 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0458fec3-51a2-44eb-9f26-8c5c3b663d74" (UID: "0458fec3-51a2-44eb-9f26-8c5c3b663d74"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:38:35.601209 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.601140 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0458fec3-51a2-44eb-9f26-8c5c3b663d74-kube-api-access-5tj6j" (OuterVolumeSpecName: "kube-api-access-5tj6j") pod "0458fec3-51a2-44eb-9f26-8c5c3b663d74" (UID: "0458fec3-51a2-44eb-9f26-8c5c3b663d74"). InnerVolumeSpecName "kube-api-access-5tj6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:38:35.699335 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.699307 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-oauth-serving-cert\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:35.699335 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.699330 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-serving-cert\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:35.699335 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.699343 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-service-ca\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:35.699528 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.699351 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-oauth-config\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:35.699528 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.699361 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0458fec3-51a2-44eb-9f26-8c5c3b663d74-console-config\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:35.699528 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:35.699370 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tj6j\" (UniqueName: \"kubernetes.io/projected/0458fec3-51a2-44eb-9f26-8c5c3b663d74-kube-api-access-5tj6j\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:38:36.148294 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.148259 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2500df92-212c-4be0-8813-972d17dc1952" path="/var/lib/kubelet/pods/2500df92-212c-4be0-8813-972d17dc1952/volumes" Sep 09 06:38:36.317200 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.317162 2563 generic.go:356] "Generic (PLEG): container finished" podID="0458fec3-51a2-44eb-9f26-8c5c3b663d74" containerID="d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2" exitCode=0 Sep 09 06:38:36.317688 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.317215 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564f69d889-5kvw4" Sep 09 06:38:36.317688 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.317253 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564f69d889-5kvw4" event={"ID":"0458fec3-51a2-44eb-9f26-8c5c3b663d74","Type":"ContainerDied","Data":"d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2"} Sep 09 06:38:36.317688 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.317305 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564f69d889-5kvw4" event={"ID":"0458fec3-51a2-44eb-9f26-8c5c3b663d74","Type":"ContainerDied","Data":"c916fc2aca908441ac18af89bec5c217ddc8ba3d12e71e9cbe296ee492e8eba0"} Sep 09 06:38:36.317688 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.317333 2563 scope.go:117] "RemoveContainer" containerID="d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2" Sep 09 06:38:36.325596 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.325578 2563 scope.go:117] "RemoveContainer" containerID="d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2" Sep 09 06:38:36.325858 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:38:36.325830 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2\": container with ID starting with d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2 not found: ID does not exist" containerID="d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2" Sep 09 06:38:36.325940 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.325859 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2"} err="failed to get container status \"d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2\": rpc error: code = NotFound desc = could not find container \"d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2\": container with ID starting with d8a8d932b7a346e7a7d5207f4500bafdbac800d60f4313e3eec264094f1df7f2 not found: ID does not exist" Sep 09 06:38:36.333908 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.333890 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564f69d889-5kvw4"] Sep 09 06:38:36.339544 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:36.339525 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-564f69d889-5kvw4"] Sep 09 06:38:38.147768 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:38:38.147733 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0458fec3-51a2-44eb-9f26-8c5c3b663d74" path="/var/lib/kubelet/pods/0458fec3-51a2-44eb-9f26-8c5c3b663d74/volumes" Sep 09 06:39:01.285482 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.285452 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ccd7c479c-h2qjh"] Sep 09 06:39:01.285913 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.285735 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0458fec3-51a2-44eb-9f26-8c5c3b663d74" containerName="console" Sep 09 06:39:01.285913 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.285746 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0458fec3-51a2-44eb-9f26-8c5c3b663d74" containerName="console" Sep 09 06:39:01.285913 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.285755 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2500df92-212c-4be0-8813-972d17dc1952" containerName="registry" Sep 09 06:39:01.285913 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.285761 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2500df92-212c-4be0-8813-972d17dc1952" containerName="registry" Sep 09 06:39:01.285913 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.285823 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0458fec3-51a2-44eb-9f26-8c5c3b663d74" containerName="console" Sep 09 06:39:01.285913 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.285833 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2500df92-212c-4be0-8813-972d17dc1952" containerName="registry" Sep 09 06:39:01.288303 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.288280 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.303888 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.303843 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ccd7c479c-h2qjh"] Sep 09 06:39:01.397337 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.397298 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-oauth-config\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.397337 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.397342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-trusted-ca-bundle\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.397563 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.397374 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-service-ca\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.397563 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.397434 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-config\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.397563 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.397469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btlfw\" (UniqueName: \"kubernetes.io/projected/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-kube-api-access-btlfw\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.397563 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.397509 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-serving-cert\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.397690 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.397568 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-oauth-serving-cert\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.498305 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.498264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-service-ca\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.498503 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.498313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-config\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.498503 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.498352 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btlfw\" (UniqueName: \"kubernetes.io/projected/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-kube-api-access-btlfw\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.498503 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.498390 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-serving-cert\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.498503 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.498423 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-oauth-serving-cert\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.498503 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.498481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-oauth-config\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.498503 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.498502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-trusted-ca-bundle\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.499466 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.499437 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-service-ca\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.499651 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.499450 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-oauth-serving-cert\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.499782 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.499670 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-trusted-ca-bundle\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.499906 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.499818 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-config\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.502150 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.502126 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-oauth-config\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.502579 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.502557 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-serving-cert\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.506758 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.506732 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btlfw\" (UniqueName: \"kubernetes.io/projected/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-kube-api-access-btlfw\") pod \"console-5ccd7c479c-h2qjh\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.599138 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.599094 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:01.876071 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:01.876022 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ccd7c479c-h2qjh"] Sep 09 06:39:01.878512 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:39:01.878483 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4beb9b6b_e05e_4169_b322_a0f5f63e6a08.slice/crio-fd50570f2a9ce7b614b0a0bd566b2dc9dc8a9f6e038daee56e7febd0d0d36b8f WatchSource:0}: Error finding container fd50570f2a9ce7b614b0a0bd566b2dc9dc8a9f6e038daee56e7febd0d0d36b8f: Status 404 returned error can't find the container with id fd50570f2a9ce7b614b0a0bd566b2dc9dc8a9f6e038daee56e7febd0d0d36b8f Sep 09 06:39:02.420978 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:02.420936 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccd7c479c-h2qjh" event={"ID":"4beb9b6b-e05e-4169-b322-a0f5f63e6a08","Type":"ContainerStarted","Data":"27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231"} Sep 09 06:39:02.421331 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:02.420998 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccd7c479c-h2qjh" event={"ID":"4beb9b6b-e05e-4169-b322-a0f5f63e6a08","Type":"ContainerStarted","Data":"fd50570f2a9ce7b614b0a0bd566b2dc9dc8a9f6e038daee56e7febd0d0d36b8f"} Sep 09 06:39:02.442451 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:02.442404 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ccd7c479c-h2qjh" podStartSLOduration=1.442384385 podStartE2EDuration="1.442384385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:39:02.440978256 +0000 UTC m=+366.765788705" watchObservedRunningTime="2026-09-09 06:39:02.442384385 +0000 UTC m=+366.767194841" Sep 09 06:39:11.599353 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:11.599313 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:11.599353 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:11.599362 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:11.604193 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:11.604173 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:12.461094 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:12.461070 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:39:12.506455 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:12.506420 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d6ff4c55-4mpfr"] Sep 09 06:39:37.527059 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.527015 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57d6ff4c55-4mpfr" podUID="ff526f80-fc41-4506-9938-e54a8a7f265c" containerName="console" containerID="cri-o://63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c" gracePeriod=15 Sep 09 06:39:37.761774 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.761753 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:39:37.853159 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853139 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-oauth-serving-cert\") pod \"ff526f80-fc41-4506-9938-e54a8a7f265c\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " Sep 09 06:39:37.853284 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853172 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rshdd\" (UniqueName: \"kubernetes.io/projected/ff526f80-fc41-4506-9938-e54a8a7f265c-kube-api-access-rshdd\") pod \"ff526f80-fc41-4506-9938-e54a8a7f265c\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " Sep 09 06:39:37.853284 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853232 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-service-ca\") pod \"ff526f80-fc41-4506-9938-e54a8a7f265c\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " Sep 09 06:39:37.853284 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853265 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-serving-cert\") pod \"ff526f80-fc41-4506-9938-e54a8a7f265c\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " Sep 09 06:39:37.853441 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853289 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-console-config\") pod \"ff526f80-fc41-4506-9938-e54a8a7f265c\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " Sep 09 06:39:37.853441 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853344 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-trusted-ca-bundle\") pod \"ff526f80-fc41-4506-9938-e54a8a7f265c\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " Sep 09 06:39:37.853441 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853368 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-oauth-config\") pod \"ff526f80-fc41-4506-9938-e54a8a7f265c\" (UID: \"ff526f80-fc41-4506-9938-e54a8a7f265c\") " Sep 09 06:39:37.853704 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853663 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ff526f80-fc41-4506-9938-e54a8a7f265c" (UID: "ff526f80-fc41-4506-9938-e54a8a7f265c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:39:37.853829 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853741 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-service-ca" (OuterVolumeSpecName: "service-ca") pod "ff526f80-fc41-4506-9938-e54a8a7f265c" (UID: "ff526f80-fc41-4506-9938-e54a8a7f265c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:39:37.853829 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853775 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-console-config" (OuterVolumeSpecName: "console-config") pod "ff526f80-fc41-4506-9938-e54a8a7f265c" (UID: "ff526f80-fc41-4506-9938-e54a8a7f265c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:39:37.853829 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.853783 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ff526f80-fc41-4506-9938-e54a8a7f265c" (UID: "ff526f80-fc41-4506-9938-e54a8a7f265c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:39:37.855748 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.855724 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ff526f80-fc41-4506-9938-e54a8a7f265c" (UID: "ff526f80-fc41-4506-9938-e54a8a7f265c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:39:37.855956 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.855925 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ff526f80-fc41-4506-9938-e54a8a7f265c" (UID: "ff526f80-fc41-4506-9938-e54a8a7f265c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:39:37.856069 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.855929 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff526f80-fc41-4506-9938-e54a8a7f265c-kube-api-access-rshdd" (OuterVolumeSpecName: "kube-api-access-rshdd") pod "ff526f80-fc41-4506-9938-e54a8a7f265c" (UID: "ff526f80-fc41-4506-9938-e54a8a7f265c"). InnerVolumeSpecName "kube-api-access-rshdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:39:37.954177 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.954159 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-service-ca\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:39:37.954177 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.954178 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-serving-cert\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:39:37.954289 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.954189 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-console-config\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:39:37.954289 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.954197 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-trusted-ca-bundle\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:39:37.954289 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.954205 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff526f80-fc41-4506-9938-e54a8a7f265c-console-oauth-config\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:39:37.954289 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.954213 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff526f80-fc41-4506-9938-e54a8a7f265c-oauth-serving-cert\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:39:37.954289 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:37.954221 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshdd\" (UniqueName: \"kubernetes.io/projected/ff526f80-fc41-4506-9938-e54a8a7f265c-kube-api-access-rshdd\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:39:38.566614 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:38.566542 2563 generic.go:356] "Generic (PLEG): container finished" podID="ff526f80-fc41-4506-9938-e54a8a7f265c" containerID="63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c" exitCode=0 Sep 09 06:39:38.566614 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:38.566584 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d6ff4c55-4mpfr" Sep 09 06:39:38.567040 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:38.566618 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d6ff4c55-4mpfr" event={"ID":"ff526f80-fc41-4506-9938-e54a8a7f265c","Type":"ContainerDied","Data":"63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c"} Sep 09 06:39:38.567040 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:38.566667 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d6ff4c55-4mpfr" event={"ID":"ff526f80-fc41-4506-9938-e54a8a7f265c","Type":"ContainerDied","Data":"f1b0fa7f07512bc96aa6ce4225e1afe331c34b4b09added239eea435cf1b6a82"} Sep 09 06:39:38.567040 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:38.566686 2563 scope.go:117] "RemoveContainer" containerID="63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c" Sep 09 06:39:38.574944 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:38.574930 2563 scope.go:117] "RemoveContainer" containerID="63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c" Sep 09 06:39:38.575217 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:39:38.575200 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c\": container with ID starting with 63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c not found: ID does not exist" containerID="63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c" Sep 09 06:39:38.575274 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:38.575225 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c"} err="failed to get container status \"63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c\": rpc error: code = NotFound desc = could not find container \"63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c\": container with ID starting with 63838931f70369f544c3fb7af1bbb7deade3988964ae2bd4762d4bcc294efa5c not found: ID does not exist" Sep 09 06:39:38.583996 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:38.583977 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d6ff4c55-4mpfr"] Sep 09 06:39:38.587543 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:38.587520 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57d6ff4c55-4mpfr"] Sep 09 06:39:40.147483 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:39:40.147448 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff526f80-fc41-4506-9938-e54a8a7f265c" path="/var/lib/kubelet/pods/ff526f80-fc41-4506-9938-e54a8a7f265c/volumes" Sep 09 06:40:36.237445 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.237368 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77959dfdb5-9hwq5"] Sep 09 06:40:36.237907 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.237644 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff526f80-fc41-4506-9938-e54a8a7f265c" containerName="console" Sep 09 06:40:36.237907 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.237654 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff526f80-fc41-4506-9938-e54a8a7f265c" containerName="console" Sep 09 06:40:36.237907 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.237710 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff526f80-fc41-4506-9938-e54a8a7f265c" containerName="console" Sep 09 06:40:36.239509 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.239492 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.253354 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.253330 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77959dfdb5-9hwq5"] Sep 09 06:40:36.271819 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.271795 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkmn5\" (UniqueName: \"kubernetes.io/projected/5330244a-89b0-47d3-8379-ff75a8320904-kube-api-access-kkmn5\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.271927 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.271827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-oauth-config\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.271927 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.271856 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-trusted-ca-bundle\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.271927 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.271871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-oauth-serving-cert\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.272095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.271930 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-service-ca\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.272095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.271954 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-console-config\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.272095 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.272045 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-serving-cert\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.373026 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkmn5\" (UniqueName: \"kubernetes.io/projected/5330244a-89b0-47d3-8379-ff75a8320904-kube-api-access-kkmn5\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.373140 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-oauth-config\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.373140 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-trusted-ca-bundle\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.373253 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-oauth-serving-cert\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.373253 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373223 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-service-ca\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.373253 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373248 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-console-config\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.373399 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373280 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-serving-cert\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.373918 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373847 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-oauth-serving-cert\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.374033 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373934 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-service-ca\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.374033 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.373940 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-trusted-ca-bundle\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.374146 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.374049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-console-config\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.375575 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.375552 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-oauth-config\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.375696 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.375680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-serving-cert\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.381279 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.381262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkmn5\" (UniqueName: \"kubernetes.io/projected/5330244a-89b0-47d3-8379-ff75a8320904-kube-api-access-kkmn5\") pod \"console-77959dfdb5-9hwq5\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.548589 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.548539 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:36.835640 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:36.835518 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77959dfdb5-9hwq5"] Sep 09 06:40:36.838629 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:40:36.838599 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5330244a_89b0_47d3_8379_ff75a8320904.slice/crio-d1cd10cd519038e27524a7f8c767e3c4de2853a76371a8485a40234a718a7dd8 WatchSource:0}: Error finding container d1cd10cd519038e27524a7f8c767e3c4de2853a76371a8485a40234a718a7dd8: Status 404 returned error can't find the container with id d1cd10cd519038e27524a7f8c767e3c4de2853a76371a8485a40234a718a7dd8 Sep 09 06:40:37.815889 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:37.815853 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77959dfdb5-9hwq5" event={"ID":"5330244a-89b0-47d3-8379-ff75a8320904","Type":"ContainerStarted","Data":"7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242"} Sep 09 06:40:37.816344 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:37.815929 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77959dfdb5-9hwq5" event={"ID":"5330244a-89b0-47d3-8379-ff75a8320904","Type":"ContainerStarted","Data":"d1cd10cd519038e27524a7f8c767e3c4de2853a76371a8485a40234a718a7dd8"} Sep 09 06:40:37.836401 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:37.836355 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77959dfdb5-9hwq5" podStartSLOduration=1.836340239 podStartE2EDuration="1.836340239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:40:37.835022698 +0000 UTC m=+462.159833155" watchObservedRunningTime="2026-09-09 06:40:37.836340239 +0000 UTC m=+462.161150695" Sep 09 06:40:46.549463 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:46.549397 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:46.550006 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:46.549522 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:46.554435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:46.554414 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:46.855693 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:46.855666 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:40:46.903296 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:40:46.903264 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ccd7c479c-h2qjh"] Sep 09 06:41:11.925431 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:11.925382 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5ccd7c479c-h2qjh" podUID="4beb9b6b-e05e-4169-b322-a0f5f63e6a08" containerName="console" containerID="cri-o://27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231" gracePeriod=15 Sep 09 06:41:12.164527 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.164499 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:41:12.239406 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.239328 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-config\") pod \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " Sep 09 06:41:12.239557 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.239411 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btlfw\" (UniqueName: \"kubernetes.io/projected/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-kube-api-access-btlfw\") pod \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " Sep 09 06:41:12.239557 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.239517 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-service-ca\") pod \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " Sep 09 06:41:12.239557 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.239534 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-oauth-serving-cert\") pod \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " Sep 09 06:41:12.239763 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.239557 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-oauth-config\") pod \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " Sep 09 06:41:12.239763 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.239586 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-trusted-ca-bundle\") pod \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " Sep 09 06:41:12.239763 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.239609 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-serving-cert\") pod \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\" (UID: \"4beb9b6b-e05e-4169-b322-a0f5f63e6a08\") " Sep 09 06:41:12.239913 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.239873 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-config" (OuterVolumeSpecName: "console-config") pod "4beb9b6b-e05e-4169-b322-a0f5f63e6a08" (UID: "4beb9b6b-e05e-4169-b322-a0f5f63e6a08"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:41:12.240052 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.240024 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-service-ca" (OuterVolumeSpecName: "service-ca") pod "4beb9b6b-e05e-4169-b322-a0f5f63e6a08" (UID: "4beb9b6b-e05e-4169-b322-a0f5f63e6a08"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:41:12.240052 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.240033 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4beb9b6b-e05e-4169-b322-a0f5f63e6a08" (UID: "4beb9b6b-e05e-4169-b322-a0f5f63e6a08"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:41:12.240052 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.240040 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4beb9b6b-e05e-4169-b322-a0f5f63e6a08" (UID: "4beb9b6b-e05e-4169-b322-a0f5f63e6a08"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:41:12.241728 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.241695 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-kube-api-access-btlfw" (OuterVolumeSpecName: "kube-api-access-btlfw") pod "4beb9b6b-e05e-4169-b322-a0f5f63e6a08" (UID: "4beb9b6b-e05e-4169-b322-a0f5f63e6a08"). InnerVolumeSpecName "kube-api-access-btlfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:41:12.241888 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.241860 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4beb9b6b-e05e-4169-b322-a0f5f63e6a08" (UID: "4beb9b6b-e05e-4169-b322-a0f5f63e6a08"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:41:12.241989 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.241937 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4beb9b6b-e05e-4169-b322-a0f5f63e6a08" (UID: "4beb9b6b-e05e-4169-b322-a0f5f63e6a08"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:41:12.340785 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.340742 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btlfw\" (UniqueName: \"kubernetes.io/projected/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-kube-api-access-btlfw\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:41:12.340785 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.340780 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-service-ca\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:41:12.340785 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.340790 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-oauth-serving-cert\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:41:12.340785 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.340798 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-oauth-config\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:41:12.341061 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.340805 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-trusted-ca-bundle\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:41:12.341061 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.340815 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-serving-cert\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:41:12.341061 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.340824 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4beb9b6b-e05e-4169-b322-a0f5f63e6a08-console-config\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:41:12.959299 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.959264 2563 generic.go:356] "Generic (PLEG): container finished" podID="4beb9b6b-e05e-4169-b322-a0f5f63e6a08" containerID="27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231" exitCode=0 Sep 09 06:41:12.959701 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.959329 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccd7c479c-h2qjh" Sep 09 06:41:12.959701 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.959352 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccd7c479c-h2qjh" event={"ID":"4beb9b6b-e05e-4169-b322-a0f5f63e6a08","Type":"ContainerDied","Data":"27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231"} Sep 09 06:41:12.959701 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.959438 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccd7c479c-h2qjh" event={"ID":"4beb9b6b-e05e-4169-b322-a0f5f63e6a08","Type":"ContainerDied","Data":"fd50570f2a9ce7b614b0a0bd566b2dc9dc8a9f6e038daee56e7febd0d0d36b8f"} Sep 09 06:41:12.959701 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.959464 2563 scope.go:117] "RemoveContainer" containerID="27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231" Sep 09 06:41:12.968331 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.968303 2563 scope.go:117] "RemoveContainer" containerID="27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231" Sep 09 06:41:12.968587 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:41:12.968566 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231\": container with ID starting with 27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231 not found: ID does not exist" containerID="27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231" Sep 09 06:41:12.968631 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.968597 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231"} err="failed to get container status \"27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231\": rpc error: code = NotFound desc = could not find container \"27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231\": container with ID starting with 27d8c5378454327e4464e2e3033cd8e201a19aa2cdce9ee4a324f7c58246f231 not found: ID does not exist" Sep 09 06:41:12.980213 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.980188 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ccd7c479c-h2qjh"] Sep 09 06:41:12.984481 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:12.984462 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ccd7c479c-h2qjh"] Sep 09 06:41:14.147347 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:41:14.147316 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4beb9b6b-e05e-4169-b322-a0f5f63e6a08" path="/var/lib/kubelet/pods/4beb9b6b-e05e-4169-b322-a0f5f63e6a08/volumes" Sep 09 06:42:49.702144 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.702106 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2"] Sep 09 06:42:49.702705 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.702545 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4beb9b6b-e05e-4169-b322-a0f5f63e6a08" containerName="console" Sep 09 06:42:49.702705 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.702562 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4beb9b6b-e05e-4169-b322-a0f5f63e6a08" containerName="console" Sep 09 06:42:49.702705 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.702659 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4beb9b6b-e05e-4169-b322-a0f5f63e6a08" containerName="console" Sep 09 06:42:49.705645 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.705624 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" Sep 09 06:42:49.707993 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.707971 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 09 06:42:49.709088 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.709068 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-4kj26\"" Sep 09 06:42:49.709183 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.709075 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 09 06:42:49.721659 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.721641 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2"] Sep 09 06:42:49.832880 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.832854 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnx6j\" (UniqueName: \"kubernetes.io/projected/f4d9cd43-c285-4dcc-9374-73c664f378b1-kube-api-access-lnx6j\") pod \"feast-operator-controller-manager-7c99664db9-k2rp2\" (UID: \"f4d9cd43-c285-4dcc-9374-73c664f378b1\") " pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" Sep 09 06:42:49.933632 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.933612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnx6j\" (UniqueName: \"kubernetes.io/projected/f4d9cd43-c285-4dcc-9374-73c664f378b1-kube-api-access-lnx6j\") pod \"feast-operator-controller-manager-7c99664db9-k2rp2\" (UID: \"f4d9cd43-c285-4dcc-9374-73c664f378b1\") " pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" Sep 09 06:42:49.942369 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:49.942351 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnx6j\" (UniqueName: \"kubernetes.io/projected/f4d9cd43-c285-4dcc-9374-73c664f378b1-kube-api-access-lnx6j\") pod \"feast-operator-controller-manager-7c99664db9-k2rp2\" (UID: \"f4d9cd43-c285-4dcc-9374-73c664f378b1\") " pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" Sep 09 06:42:50.015328 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:50.015276 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" Sep 09 06:42:50.285884 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:50.285761 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2"] Sep 09 06:42:50.288433 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:42:50.288396 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d9cd43_c285_4dcc_9374_73c664f378b1.slice/crio-a548b0ee476231f0334c7da4518ca988b0eea3df3d8b4e5cc6130d3c6e5bc8ae WatchSource:0}: Error finding container a548b0ee476231f0334c7da4518ca988b0eea3df3d8b4e5cc6130d3c6e5bc8ae: Status 404 returned error can't find the container with id a548b0ee476231f0334c7da4518ca988b0eea3df3d8b4e5cc6130d3c6e5bc8ae Sep 09 06:42:50.290500 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:50.290485 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 06:42:50.336524 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:50.336497 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" event={"ID":"f4d9cd43-c285-4dcc-9374-73c664f378b1","Type":"ContainerStarted","Data":"a548b0ee476231f0334c7da4518ca988b0eea3df3d8b4e5cc6130d3c6e5bc8ae"} Sep 09 06:42:51.515981 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.515926 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fdb6b786f-7lws7"] Sep 09 06:42:51.522596 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.522569 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.535746 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.535722 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdb6b786f-7lws7"] Sep 09 06:42:51.648203 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.648135 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e6a14f8-7d19-4d94-91cf-dd9beb024918-console-oauth-config\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.648203 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.648184 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-oauth-serving-cert\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.648442 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.648231 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-trusted-ca-bundle\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.648442 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.648256 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmktj\" (UniqueName: \"kubernetes.io/projected/0e6a14f8-7d19-4d94-91cf-dd9beb024918-kube-api-access-zmktj\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.648442 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.648317 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-console-config\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.648442 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.648358 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e6a14f8-7d19-4d94-91cf-dd9beb024918-console-serving-cert\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.648609 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.648508 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-service-ca\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.748909 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.748881 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e6a14f8-7d19-4d94-91cf-dd9beb024918-console-oauth-config\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.749110 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.748914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-oauth-serving-cert\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.749110 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.748978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-trusted-ca-bundle\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.749110 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.749006 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmktj\" (UniqueName: \"kubernetes.io/projected/0e6a14f8-7d19-4d94-91cf-dd9beb024918-kube-api-access-zmktj\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.749110 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.749033 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-console-config\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.749110 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.749058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e6a14f8-7d19-4d94-91cf-dd9beb024918-console-serving-cert\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.749110 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.749094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-service-ca\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.749780 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.749724 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-oauth-serving-cert\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.749780 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.749724 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-service-ca\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.750261 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.750232 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-trusted-ca-bundle\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.750365 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.750295 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e6a14f8-7d19-4d94-91cf-dd9beb024918-console-config\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.752862 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.752819 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e6a14f8-7d19-4d94-91cf-dd9beb024918-console-oauth-config\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.753568 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.753546 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e6a14f8-7d19-4d94-91cf-dd9beb024918-console-serving-cert\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.762322 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.762282 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmktj\" (UniqueName: \"kubernetes.io/projected/0e6a14f8-7d19-4d94-91cf-dd9beb024918-kube-api-access-zmktj\") pod \"console-5fdb6b786f-7lws7\" (UID: \"0e6a14f8-7d19-4d94-91cf-dd9beb024918\") " pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:51.834148 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:51.834084 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:42:52.726148 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:52.726079 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdb6b786f-7lws7"] Sep 09 06:42:52.728822 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:42:52.728800 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6a14f8_7d19_4d94_91cf_dd9beb024918.slice/crio-65e522485a505d7fd6ceb57128a771e12d8c4d8736053ea69d9a663a69c1ac0e WatchSource:0}: Error finding container 65e522485a505d7fd6ceb57128a771e12d8c4d8736053ea69d9a663a69c1ac0e: Status 404 returned error can't find the container with id 65e522485a505d7fd6ceb57128a771e12d8c4d8736053ea69d9a663a69c1ac0e Sep 09 06:42:53.353914 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:53.353877 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdb6b786f-7lws7" event={"ID":"0e6a14f8-7d19-4d94-91cf-dd9beb024918","Type":"ContainerStarted","Data":"f8722c9b73a94b67fa55675e1c41a565e95c4319d4a7fa507a89f451d843ab65"} Sep 09 06:42:53.354168 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:53.353930 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdb6b786f-7lws7" event={"ID":"0e6a14f8-7d19-4d94-91cf-dd9beb024918","Type":"ContainerStarted","Data":"65e522485a505d7fd6ceb57128a771e12d8c4d8736053ea69d9a663a69c1ac0e"} Sep 09 06:42:53.355367 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:53.355341 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" event={"ID":"f4d9cd43-c285-4dcc-9374-73c664f378b1","Type":"ContainerStarted","Data":"1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3"} Sep 09 06:42:53.355466 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:53.355401 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" Sep 09 06:42:53.372144 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:53.372091 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fdb6b786f-7lws7" podStartSLOduration=2.372078275 podStartE2EDuration="2.372078275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:42:53.371104814 +0000 UTC m=+597.695915271" watchObservedRunningTime="2026-09-09 06:42:53.372078275 +0000 UTC m=+597.696888731" Sep 09 06:42:53.387569 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:42:53.387525 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" podStartSLOduration=2.170017607 podStartE2EDuration="4.387507282s" totalImagesPullingTime="2.217489675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:42:53.386481271 +0000 UTC m=+597.711291729" watchObservedRunningTime="2026-09-09 06:42:53.387507282 +0000 UTC m=+597.712317739" Sep 09 06:43:01.834718 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:01.834679 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:43:01.835195 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:01.834761 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:43:01.839542 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:01.839523 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:43:02.394232 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:02.394206 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdb6b786f-7lws7" Sep 09 06:43:02.441180 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:02.441147 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77959dfdb5-9hwq5"] Sep 09 06:43:04.361874 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:04.361849 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" Sep 09 06:43:27.383016 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.382930 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd"] Sep 09 06:43:27.390333 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.390313 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.393466 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.393446 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 06:43:27.393554 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.393448 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 06:43:27.393801 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.393774 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 06:43:27.393918 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.393814 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 06:43:27.393918 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.393829 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 06:43:27.393918 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.393851 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-k7p5n\"" Sep 09 06:43:27.393918 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.393821 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 06:43:27.394204 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.394188 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 09 06:43:27.399180 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.399159 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd"] Sep 09 06:43:27.425184 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.425163 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/20e49aab-bcaf-4b71-ad51-d023791ca219-feast-data\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.425288 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.425193 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-offline-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.425288 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.425237 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-ui-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.425288 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.425265 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-registry-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.425424 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.425330 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkfv\" (UniqueName: \"kubernetes.io/projected/20e49aab-bcaf-4b71-ad51-d023791ca219-kube-api-access-9qkfv\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.425424 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.425364 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-online-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.467534 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.467502 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77959dfdb5-9hwq5" podUID="5330244a-89b0-47d3-8379-ff75a8320904" containerName="console" containerID="cri-o://7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242" gracePeriod=15 Sep 09 06:43:27.526561 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.526536 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-online-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.526648 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.526568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/20e49aab-bcaf-4b71-ad51-d023791ca219-feast-data\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.526648 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.526588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-offline-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.526648 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.526629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-ui-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.526805 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.526659 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-registry-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.526805 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:43:27.526678 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 06:43:27.526805 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:43:27.526763 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-online-tls podName:20e49aab-bcaf-4b71-ad51-d023791ca219 nodeName:}" failed. No retries permitted until 2026-09-09 06:43:28.026742445 +0000 UTC m=+632.351552879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-online-tls") pod "feast-simple-feast-setup-6b94879c9f-bnfsd" (UID: "20e49aab-bcaf-4b71-ad51-d023791ca219") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 06:43:27.526805 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:43:27.526764 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 06:43:27.526805 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.526797 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkfv\" (UniqueName: \"kubernetes.io/projected/20e49aab-bcaf-4b71-ad51-d023791ca219-kube-api-access-9qkfv\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.527058 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:43:27.526844 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-ui-tls podName:20e49aab-bcaf-4b71-ad51-d023791ca219 nodeName:}" failed. No retries permitted until 2026-09-09 06:43:28.026825668 +0000 UTC m=+632.351636124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-ui-tls") pod "feast-simple-feast-setup-6b94879c9f-bnfsd" (UID: "20e49aab-bcaf-4b71-ad51-d023791ca219") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 06:43:27.527058 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.526948 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/20e49aab-bcaf-4b71-ad51-d023791ca219-feast-data\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.529044 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.529028 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-registry-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.529218 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.529202 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-offline-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.535766 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.535744 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkfv\" (UniqueName: \"kubernetes.io/projected/20e49aab-bcaf-4b71-ad51-d023791ca219-kube-api-access-9qkfv\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:27.709384 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.709356 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:43:27.728690 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.728671 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-console-config\") pod \"5330244a-89b0-47d3-8379-ff75a8320904\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " Sep 09 06:43:27.728841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.728710 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkmn5\" (UniqueName: \"kubernetes.io/projected/5330244a-89b0-47d3-8379-ff75a8320904-kube-api-access-kkmn5\") pod \"5330244a-89b0-47d3-8379-ff75a8320904\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " Sep 09 06:43:27.728841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.728777 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-oauth-config\") pod \"5330244a-89b0-47d3-8379-ff75a8320904\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " Sep 09 06:43:27.728841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.728800 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-service-ca\") pod \"5330244a-89b0-47d3-8379-ff75a8320904\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " Sep 09 06:43:27.728841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.728830 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-trusted-ca-bundle\") pod \"5330244a-89b0-47d3-8379-ff75a8320904\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " Sep 09 06:43:27.729114 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.728860 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-oauth-serving-cert\") pod \"5330244a-89b0-47d3-8379-ff75a8320904\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " Sep 09 06:43:27.729114 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.728897 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-serving-cert\") pod \"5330244a-89b0-47d3-8379-ff75a8320904\" (UID: \"5330244a-89b0-47d3-8379-ff75a8320904\") " Sep 09 06:43:27.729114 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.729036 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-console-config" (OuterVolumeSpecName: "console-config") pod "5330244a-89b0-47d3-8379-ff75a8320904" (UID: "5330244a-89b0-47d3-8379-ff75a8320904"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:43:27.729266 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.729187 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-console-config\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:43:27.729356 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.729246 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-service-ca" (OuterVolumeSpecName: "service-ca") pod "5330244a-89b0-47d3-8379-ff75a8320904" (UID: "5330244a-89b0-47d3-8379-ff75a8320904"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:43:27.729493 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.729463 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5330244a-89b0-47d3-8379-ff75a8320904" (UID: "5330244a-89b0-47d3-8379-ff75a8320904"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:43:27.729636 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.729611 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5330244a-89b0-47d3-8379-ff75a8320904" (UID: "5330244a-89b0-47d3-8379-ff75a8320904"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:43:27.731518 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.731497 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5330244a-89b0-47d3-8379-ff75a8320904" (UID: "5330244a-89b0-47d3-8379-ff75a8320904"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:43:27.731728 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.731708 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5330244a-89b0-47d3-8379-ff75a8320904-kube-api-access-kkmn5" (OuterVolumeSpecName: "kube-api-access-kkmn5") pod "5330244a-89b0-47d3-8379-ff75a8320904" (UID: "5330244a-89b0-47d3-8379-ff75a8320904"). InnerVolumeSpecName "kube-api-access-kkmn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:43:27.731728 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.731709 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5330244a-89b0-47d3-8379-ff75a8320904" (UID: "5330244a-89b0-47d3-8379-ff75a8320904"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:43:27.829806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.829782 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkmn5\" (UniqueName: \"kubernetes.io/projected/5330244a-89b0-47d3-8379-ff75a8320904-kube-api-access-kkmn5\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:43:27.829806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.829802 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-oauth-config\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:43:27.829806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.829811 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-service-ca\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:43:27.830004 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.829819 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-trusted-ca-bundle\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:43:27.830004 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.829827 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5330244a-89b0-47d3-8379-ff75a8320904-oauth-serving-cert\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:43:27.830004 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:27.829836 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5330244a-89b0-47d3-8379-ff75a8320904-console-serving-cert\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:43:28.030784 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.030706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-ui-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:28.030784 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.030766 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-online-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:28.033197 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.033169 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-ui-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:28.033306 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.033227 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-online-tls\") pod \"feast-simple-feast-setup-6b94879c9f-bnfsd\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:28.310476 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.310396 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:43:28.490888 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.490860 2563 generic.go:356] "Generic (PLEG): container finished" podID="5330244a-89b0-47d3-8379-ff75a8320904" containerID="7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242" exitCode=0 Sep 09 06:43:28.491233 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.490916 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77959dfdb5-9hwq5" Sep 09 06:43:28.491233 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.490946 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77959dfdb5-9hwq5" event={"ID":"5330244a-89b0-47d3-8379-ff75a8320904","Type":"ContainerDied","Data":"7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242"} Sep 09 06:43:28.491233 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.491058 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77959dfdb5-9hwq5" event={"ID":"5330244a-89b0-47d3-8379-ff75a8320904","Type":"ContainerDied","Data":"d1cd10cd519038e27524a7f8c767e3c4de2853a76371a8485a40234a718a7dd8"} Sep 09 06:43:28.491233 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.491083 2563 scope.go:117] "RemoveContainer" containerID="7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242" Sep 09 06:43:28.499448 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.499432 2563 scope.go:117] "RemoveContainer" containerID="7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242" Sep 09 06:43:28.499713 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:43:28.499696 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242\": container with ID starting with 7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242 not found: ID does not exist" containerID="7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242" Sep 09 06:43:28.499758 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.499722 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242"} err="failed to get container status \"7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242\": rpc error: code = NotFound desc = could not find container \"7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242\": container with ID starting with 7ea9818fa6a9e7ff6c1641545626354a9016d01a71cb3b191004214860b0a242 not found: ID does not exist" Sep 09 06:43:28.508949 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.508929 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77959dfdb5-9hwq5"] Sep 09 06:43:28.512726 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.512706 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77959dfdb5-9hwq5"] Sep 09 06:43:28.588442 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:28.588382 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd"] Sep 09 06:43:28.592195 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:43:28.592171 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20e49aab_bcaf_4b71_ad51_d023791ca219.slice/crio-4959cc9faf41e5c0db05591c6169f0e0a18e17dda31f408ca3e657b7e27fcdfb WatchSource:0}: Error finding container 4959cc9faf41e5c0db05591c6169f0e0a18e17dda31f408ca3e657b7e27fcdfb: Status 404 returned error can't find the container with id 4959cc9faf41e5c0db05591c6169f0e0a18e17dda31f408ca3e657b7e27fcdfb Sep 09 06:43:29.501475 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:29.501401 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerStarted","Data":"4959cc9faf41e5c0db05591c6169f0e0a18e17dda31f408ca3e657b7e27fcdfb"} Sep 09 06:43:30.148426 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:30.148397 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5330244a-89b0-47d3-8379-ff75a8320904" path="/var/lib/kubelet/pods/5330244a-89b0-47d3-8379-ff75a8320904/volumes" Sep 09 06:43:52.606070 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:52.606029 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerStarted","Data":"079c40cd52478fab3152d02bb6c6777aa479e69bc07066efadcf71e84f21fb03"} Sep 09 06:43:57.627633 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:57.627601 2563 generic.go:356] "Generic (PLEG): container finished" podID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerID="079c40cd52478fab3152d02bb6c6777aa479e69bc07066efadcf71e84f21fb03" exitCode=0 Sep 09 06:43:57.628026 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:57.627636 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerDied","Data":"079c40cd52478fab3152d02bb6c6777aa479e69bc07066efadcf71e84f21fb03"} Sep 09 06:43:58.633412 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:43:58.633371 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerStarted","Data":"317aba68862b82d741c2adffc63c60a2ae4b3063e9ad1943c496aab6f6662c2c"} Sep 09 06:44:04.666359 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:04.666325 2563 generic.go:356] "Generic (PLEG): container finished" podID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerID="317aba68862b82d741c2adffc63c60a2ae4b3063e9ad1943c496aab6f6662c2c" exitCode=0 Sep 09 06:44:04.666756 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:04.666396 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerDied","Data":"317aba68862b82d741c2adffc63c60a2ae4b3063e9ad1943c496aab6f6662c2c"} Sep 09 06:44:04.675796 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:04.675777 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 09 06:44:05.677661 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:05.677621 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerStarted","Data":"d719118b9dbed7bdc67a5c3a57be22edabb8d12a3d6cc18119e73161b61db50c"} Sep 09 06:44:05.678182 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:05.677693 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerStarted","Data":"ec2ff2db48a6543062bcded151a75745033ab41d6b0f6aceffd136431e161b5c"} Sep 09 06:44:05.678182 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:05.677717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerStarted","Data":"df757099687c6703981327aa3471a7caf611e09b25329f89598eb4e5524973f7"} Sep 09 06:44:05.678182 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:05.677733 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerStarted","Data":"33987cfca0b93cd81454e565432ec4fd90050c2f7a2ddb212173a2cd5b16d23b"} Sep 09 06:44:05.712979 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:05.712892 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podStartSLOduration=14.893502771 podStartE2EDuration="38.712867201s" totalImagesPullingTime="23.81936443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:44:05.711234353 +0000 UTC m=+670.036044810" watchObservedRunningTime="2026-09-09 06:44:05.712867201 +0000 UTC m=+670.037677659" Sep 09 06:44:07.311334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:07.311233 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:07.311334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:07.311298 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:07.311334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:07.311318 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:07.311334 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:07.311339 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:07.313328 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:07.313292 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused" Sep 09 06:44:07.313471 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:07.313343 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 06:44:07.313471 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:07.313406 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 06:44:07.313599 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:07.313345 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Sep 09 06:44:08.311137 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:08.311087 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:08.311330 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:08.311163 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:08.311330 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:08.311184 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:08.311330 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:08.311202 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:10.311197 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:10.311157 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Sep 09 06:44:10.311900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:10.311183 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 06:44:10.312009 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:10.311178 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused" Sep 09 06:44:10.312009 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:10.311447 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 06:44:13.311751 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:13.311718 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:13.312200 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:13.312010 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:13.312200 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:13.312146 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:13.312760 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:13.312735 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:13.313041 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:13.313022 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:13.313137 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:13.313089 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:13.317274 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:13.317252 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:44:13.723236 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:44:13.723210 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:45:15.677332 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:15.677292 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd"] Sep 09 06:45:15.677852 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:15.677588 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="registry" containerID="cri-o://33987cfca0b93cd81454e565432ec4fd90050c2f7a2ddb212173a2cd5b16d23b" gracePeriod=30 Sep 09 06:45:15.677852 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:15.677637 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="online" containerID="cri-o://df757099687c6703981327aa3471a7caf611e09b25329f89598eb4e5524973f7" gracePeriod=30 Sep 09 06:45:15.677852 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:15.677690 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="ui" containerID="cri-o://d719118b9dbed7bdc67a5c3a57be22edabb8d12a3d6cc18119e73161b61db50c" gracePeriod=30 Sep 09 06:45:15.677852 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:15.677643 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="offline" containerID="cri-o://ec2ff2db48a6543062bcded151a75745033ab41d6b0f6aceffd136431e161b5c" gracePeriod=30 Sep 09 06:45:16.140693 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.140661 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl"] Sep 09 06:45:16.141137 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.141118 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5330244a-89b0-47d3-8379-ff75a8320904" containerName="console" Sep 09 06:45:16.141137 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.141139 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5330244a-89b0-47d3-8379-ff75a8320904" containerName="console" Sep 09 06:45:16.141304 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.141227 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5330244a-89b0-47d3-8379-ff75a8320904" containerName="console" Sep 09 06:45:16.145068 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.145040 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.157076 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.157052 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-kccs7\"" Sep 09 06:45:16.166324 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.166300 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl"] Sep 09 06:45:16.287869 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.287839 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-registry-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.288047 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.287890 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a892fcd7-f82c-4a28-ba48-d79e8b11675c-feast-data\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.288047 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.287915 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-online-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.288047 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.287939 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsn4j\" (UniqueName: \"kubernetes.io/projected/a892fcd7-f82c-4a28-ba48-d79e8b11675c-kube-api-access-bsn4j\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.288047 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.287983 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-offline-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.288047 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.288006 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-ui-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.389146 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.389116 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-registry-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.389146 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.389153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a892fcd7-f82c-4a28-ba48-d79e8b11675c-feast-data\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.389380 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.389172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-online-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.389380 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.389190 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsn4j\" (UniqueName: \"kubernetes.io/projected/a892fcd7-f82c-4a28-ba48-d79e8b11675c-kube-api-access-bsn4j\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.389380 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.389209 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-offline-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.389380 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.389353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-ui-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.389576 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:45:16.389490 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 06:45:16.389576 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:45:16.389564 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-ui-tls podName:a892fcd7-f82c-4a28-ba48-d79e8b11675c nodeName:}" failed. No retries permitted until 2026-09-09 06:45:16.889541374 +0000 UTC m=+741.214351827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-ui-tls") pod "feast-simple-feast-setup-6b94879c9f-gdfhl" (UID: "a892fcd7-f82c-4a28-ba48-d79e8b11675c") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 06:45:16.389728 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.389579 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a892fcd7-f82c-4a28-ba48-d79e8b11675c-feast-data\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.391856 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.391802 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-registry-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.391950 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.391864 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-online-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.391950 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.391916 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-offline-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.398537 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.398516 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsn4j\" (UniqueName: \"kubernetes.io/projected/a892fcd7-f82c-4a28-ba48-d79e8b11675c-kube-api-access-bsn4j\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.893242 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.893206 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-ui-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.895616 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.895594 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-ui-tls\") pod \"feast-simple-feast-setup-6b94879c9f-gdfhl\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:16.985026 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.984995 2563 generic.go:356] "Generic (PLEG): container finished" podID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerID="d719118b9dbed7bdc67a5c3a57be22edabb8d12a3d6cc18119e73161b61db50c" exitCode=0 Sep 09 06:45:16.985026 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.985022 2563 generic.go:356] "Generic (PLEG): container finished" podID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerID="df757099687c6703981327aa3471a7caf611e09b25329f89598eb4e5524973f7" exitCode=0 Sep 09 06:45:16.985189 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.985076 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerDied","Data":"d719118b9dbed7bdc67a5c3a57be22edabb8d12a3d6cc18119e73161b61db50c"} Sep 09 06:45:16.985189 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:16.985125 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerDied","Data":"df757099687c6703981327aa3471a7caf611e09b25329f89598eb4e5524973f7"} Sep 09 06:45:17.055363 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:17.055341 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:17.332874 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:17.332841 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl"] Sep 09 06:45:17.333339 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:45:17.333312 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda892fcd7_f82c_4a28_ba48_d79e8b11675c.slice/crio-a479a6271e293b0c34667a3ed5071f0d9477601007c4fe4a325c6828f7f84a0d WatchSource:0}: Error finding container a479a6271e293b0c34667a3ed5071f0d9477601007c4fe4a325c6828f7f84a0d: Status 404 returned error can't find the container with id a479a6271e293b0c34667a3ed5071f0d9477601007c4fe4a325c6828f7f84a0d Sep 09 06:45:17.990584 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:17.990550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerStarted","Data":"ee864b6837037117170d2b711ba02b19dfec6bae934b6c1fc022c0ac9e6e6504"} Sep 09 06:45:17.991058 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:17.990597 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerStarted","Data":"a479a6271e293b0c34667a3ed5071f0d9477601007c4fe4a325c6828f7f84a0d"} Sep 09 06:45:23.011952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:23.011920 2563 generic.go:356] "Generic (PLEG): container finished" podID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerID="ee864b6837037117170d2b711ba02b19dfec6bae934b6c1fc022c0ac9e6e6504" exitCode=0 Sep 09 06:45:23.012359 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:23.011995 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerDied","Data":"ee864b6837037117170d2b711ba02b19dfec6bae934b6c1fc022c0ac9e6e6504"} Sep 09 06:45:23.313269 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:23.313155 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 06:45:23.720876 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:23.720819 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 06:45:24.019313 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:24.019222 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerStarted","Data":"f49c50c406cf66b90e86772829b09e3c08694b39c8e8b16b417e3ea4a628ad09"} Sep 09 06:45:29.045206 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:29.045171 2563 generic.go:356] "Generic (PLEG): container finished" podID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerID="f49c50c406cf66b90e86772829b09e3c08694b39c8e8b16b417e3ea4a628ad09" exitCode=0 Sep 09 06:45:29.045654 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:29.045218 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerDied","Data":"f49c50c406cf66b90e86772829b09e3c08694b39c8e8b16b417e3ea4a628ad09"} Sep 09 06:45:30.056865 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:30.056823 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerStarted","Data":"16c63059a002c0ddead680d6247946edb3e5bd615ae60c7425b1399438352a8b"} Sep 09 06:45:30.057393 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:30.056906 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerStarted","Data":"0154628ea18c57355e414a2dd49c1f579b06394754193a0bd038b37a13b1a949"} Sep 09 06:45:30.057393 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:30.056927 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerStarted","Data":"5fc6826c5eec4a125b25e2fced03ddae5dd250e0e66da2f866b0d31b7496dd48"} Sep 09 06:45:30.057393 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:30.056984 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerStarted","Data":"b922b95cdc2e30f6d13d9b45b0bd195f764fb39352adf65576c3e766002be55f"} Sep 09 06:45:30.089115 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:30.089043 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podStartSLOduration=14.089019947 podStartE2EDuration="14.089019947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:45:30.087030469 +0000 UTC m=+754.411840925" watchObservedRunningTime="2026-09-09 06:45:30.089019947 +0000 UTC m=+754.413830405" Sep 09 06:45:32.055639 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:32.055608 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:32.056330 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:32.056310 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:32.056479 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:32.056456 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:32.056605 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:32.056486 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:32.058524 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:32.058281 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Sep 09 06:45:32.058524 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:32.058330 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 06:45:32.058524 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:32.058330 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" Sep 09 06:45:32.058524 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:32.058384 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 06:45:33.313275 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:33.312934 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 06:45:33.721452 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:33.721399 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 06:45:35.056296 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:35.056241 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 06:45:35.056797 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:35.056242 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" Sep 09 06:45:35.056797 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:35.056241 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Sep 09 06:45:35.056797 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:35.056255 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 06:45:37.056091 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:37.056053 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:37.056556 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:37.056114 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:37.056556 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:37.056134 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:37.056556 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:37.056152 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:38.056888 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:38.056854 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:38.057294 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:38.057203 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:38.057634 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:38.057612 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:38.057743 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:38.057728 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:38.062083 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:38.062066 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:38.107830 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:38.107811 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:38.107992 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:38.107955 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:38.110472 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:38.110451 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:45:43.312738 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:43.312691 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Sep 09 06:45:43.313248 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:43.312829 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:45:43.720323 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:43.720275 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Sep 09 06:45:43.720491 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:43.720415 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:45:46.145134 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.145107 2563 generic.go:356] "Generic (PLEG): container finished" podID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerID="ec2ff2db48a6543062bcded151a75745033ab41d6b0f6aceffd136431e161b5c" exitCode=137 Sep 09 06:45:46.145134 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.145130 2563 generic.go:356] "Generic (PLEG): container finished" podID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerID="33987cfca0b93cd81454e565432ec4fd90050c2f7a2ddb212173a2cd5b16d23b" exitCode=137 Sep 09 06:45:46.146839 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.146817 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerDied","Data":"ec2ff2db48a6543062bcded151a75745033ab41d6b0f6aceffd136431e161b5c"} Sep 09 06:45:46.146917 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.146855 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerDied","Data":"33987cfca0b93cd81454e565432ec4fd90050c2f7a2ddb212173a2cd5b16d23b"} Sep 09 06:45:46.324622 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.324598 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:45:46.344768 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.344745 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/20e49aab-bcaf-4b71-ad51-d023791ca219-feast-data\") pod \"20e49aab-bcaf-4b71-ad51-d023791ca219\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " Sep 09 06:45:46.344857 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.344803 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qkfv\" (UniqueName: \"kubernetes.io/projected/20e49aab-bcaf-4b71-ad51-d023791ca219-kube-api-access-9qkfv\") pod \"20e49aab-bcaf-4b71-ad51-d023791ca219\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " Sep 09 06:45:46.344857 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.344836 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-online-tls\") pod \"20e49aab-bcaf-4b71-ad51-d023791ca219\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " Sep 09 06:45:46.344924 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.344854 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-offline-tls\") pod \"20e49aab-bcaf-4b71-ad51-d023791ca219\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " Sep 09 06:45:46.344924 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.344893 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-ui-tls\") pod \"20e49aab-bcaf-4b71-ad51-d023791ca219\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " Sep 09 06:45:46.345038 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.344992 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-registry-tls\") pod \"20e49aab-bcaf-4b71-ad51-d023791ca219\" (UID: \"20e49aab-bcaf-4b71-ad51-d023791ca219\") " Sep 09 06:45:46.345296 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.345267 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e49aab-bcaf-4b71-ad51-d023791ca219-feast-data" (OuterVolumeSpecName: "feast-data") pod "20e49aab-bcaf-4b71-ad51-d023791ca219" (UID: "20e49aab-bcaf-4b71-ad51-d023791ca219"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:45:46.347384 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.347343 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "20e49aab-bcaf-4b71-ad51-d023791ca219" (UID: "20e49aab-bcaf-4b71-ad51-d023791ca219"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:45:46.347384 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.347372 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "20e49aab-bcaf-4b71-ad51-d023791ca219" (UID: "20e49aab-bcaf-4b71-ad51-d023791ca219"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:45:46.347535 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.347434 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e49aab-bcaf-4b71-ad51-d023791ca219-kube-api-access-9qkfv" (OuterVolumeSpecName: "kube-api-access-9qkfv") pod "20e49aab-bcaf-4b71-ad51-d023791ca219" (UID: "20e49aab-bcaf-4b71-ad51-d023791ca219"). InnerVolumeSpecName "kube-api-access-9qkfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:45:46.347801 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.347760 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "20e49aab-bcaf-4b71-ad51-d023791ca219" (UID: "20e49aab-bcaf-4b71-ad51-d023791ca219"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:45:46.347891 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.347778 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-online-tls" (OuterVolumeSpecName: "online-tls") pod "20e49aab-bcaf-4b71-ad51-d023791ca219" (UID: "20e49aab-bcaf-4b71-ad51-d023791ca219"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:45:46.445855 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.445797 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/20e49aab-bcaf-4b71-ad51-d023791ca219-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:45:46.445855 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.445821 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkfv\" (UniqueName: \"kubernetes.io/projected/20e49aab-bcaf-4b71-ad51-d023791ca219-kube-api-access-9qkfv\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:45:46.445855 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.445830 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:45:46.445855 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.445840 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:45:46.445855 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.445847 2563 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-ui-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:45:46.445855 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:46.445854 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/20e49aab-bcaf-4b71-ad51-d023791ca219-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:45:47.152650 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.152617 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" event={"ID":"20e49aab-bcaf-4b71-ad51-d023791ca219","Type":"ContainerDied","Data":"4959cc9faf41e5c0db05591c6169f0e0a18e17dda31f408ca3e657b7e27fcdfb"} Sep 09 06:45:47.152650 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.152639 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd" Sep 09 06:45:47.153108 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.152671 2563 scope.go:117] "RemoveContainer" containerID="d719118b9dbed7bdc67a5c3a57be22edabb8d12a3d6cc18119e73161b61db50c" Sep 09 06:45:47.162030 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.162014 2563 scope.go:117] "RemoveContainer" containerID="ec2ff2db48a6543062bcded151a75745033ab41d6b0f6aceffd136431e161b5c" Sep 09 06:45:47.169366 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.169348 2563 scope.go:117] "RemoveContainer" containerID="df757099687c6703981327aa3471a7caf611e09b25329f89598eb4e5524973f7" Sep 09 06:45:47.175913 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.175892 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd"] Sep 09 06:45:47.177192 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.177174 2563 scope.go:117] "RemoveContainer" containerID="33987cfca0b93cd81454e565432ec4fd90050c2f7a2ddb212173a2cd5b16d23b" Sep 09 06:45:47.180113 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.180095 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-bnfsd"] Sep 09 06:45:47.185197 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.185178 2563 scope.go:117] "RemoveContainer" containerID="317aba68862b82d741c2adffc63c60a2ae4b3063e9ad1943c496aab6f6662c2c" Sep 09 06:45:47.197566 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:47.197546 2563 scope.go:117] "RemoveContainer" containerID="079c40cd52478fab3152d02bb6c6777aa479e69bc07066efadcf71e84f21fb03" Sep 09 06:45:48.147094 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:45:48.147061 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" path="/var/lib/kubelet/pods/20e49aab-bcaf-4b71-ad51-d023791ca219/volumes" Sep 09 06:46:40.998447 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998367 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x"] Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998722 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="feast-init" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998736 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="feast-init" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998745 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="ui" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998751 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="ui" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998759 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="feast-apply" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998764 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="feast-apply" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998776 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="registry" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998781 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="registry" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998787 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="offline" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998792 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="offline" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998799 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="online" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998803 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="online" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998856 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="online" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998864 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="ui" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998871 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="registry" Sep 09 06:46:40.998890 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:40.998877 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e49aab-bcaf-4b71-ad51-d023791ca219" containerName="offline" Sep 09 06:46:41.002445 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.002425 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.005077 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.005055 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-vlbd6\"" Sep 09 06:46:41.005207 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.005055 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 06:46:41.005306 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.005273 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 09 06:46:41.005485 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.005470 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 06:46:41.006093 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.006078 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 06:46:41.006166 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.006124 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 06:46:41.006250 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.006232 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 06:46:41.006528 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.006510 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 06:46:41.015206 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.015185 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x"] Sep 09 06:46:41.144180 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.144143 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-offline-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.144371 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.144192 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-ui-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.144371 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.144257 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/daff82a0-e8d6-486b-a229-23bc25cb4be8-feast-data\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.144371 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.144293 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-online-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.144371 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.144339 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/daff82a0-e8d6-486b-a229-23bc25cb4be8-registry-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.144371 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.144365 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9pp\" (UniqueName: \"kubernetes.io/projected/daff82a0-e8d6-486b-a229-23bc25cb4be8-kube-api-access-js9pp\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.245183 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.245149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js9pp\" (UniqueName: \"kubernetes.io/projected/daff82a0-e8d6-486b-a229-23bc25cb4be8-kube-api-access-js9pp\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.245356 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.245228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-offline-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.245356 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.245268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-ui-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.245356 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.245317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/daff82a0-e8d6-486b-a229-23bc25cb4be8-feast-data\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.245356 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.245347 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-online-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.245558 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.245375 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/daff82a0-e8d6-486b-a229-23bc25cb4be8-registry-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.245765 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.245735 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/daff82a0-e8d6-486b-a229-23bc25cb4be8-feast-data\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.246148 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.246125 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/daff82a0-e8d6-486b-a229-23bc25cb4be8-registry-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.247932 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.247913 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-offline-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.248051 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.247940 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-ui-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.248051 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.247989 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-online-tls\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.253051 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.252998 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9pp\" (UniqueName: \"kubernetes.io/projected/daff82a0-e8d6-486b-a229-23bc25cb4be8-kube-api-access-js9pp\") pod \"feast-simple-feast-remote-setup-85bf6597c5-z7l2x\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.311821 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.311789 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:46:41.588782 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:41.588752 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x"] Sep 09 06:46:41.591345 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:46:41.591321 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaff82a0_e8d6_486b_a229_23bc25cb4be8.slice/crio-e89c424867a8212159c55cca81c59e8374e82a4dc49288626ef8243f3104f27f WatchSource:0}: Error finding container e89c424867a8212159c55cca81c59e8374e82a4dc49288626ef8243f3104f27f: Status 404 returned error can't find the container with id e89c424867a8212159c55cca81c59e8374e82a4dc49288626ef8243f3104f27f Sep 09 06:46:42.385848 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:42.385818 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerStarted","Data":"4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780"} Sep 09 06:46:42.385848 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:42.385857 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerStarted","Data":"e89c424867a8212159c55cca81c59e8374e82a4dc49288626ef8243f3104f27f"} Sep 09 06:46:46.405524 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:46.405496 2563 generic.go:356] "Generic (PLEG): container finished" podID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerID="4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780" exitCode=0 Sep 09 06:46:46.405885 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:46.405569 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerDied","Data":"4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780"} Sep 09 06:46:47.411841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:46:47.411803 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerStarted","Data":"2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a"} Sep 09 06:47:07.497179 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:07.497143 2563 generic.go:356] "Generic (PLEG): container finished" podID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerID="2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a" exitCode=0 Sep 09 06:47:07.497551 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:07.497215 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerDied","Data":"2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a"} Sep 09 06:47:07.500812 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:07.500792 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 09 06:47:08.507003 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:08.506944 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerStarted","Data":"26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667"} Sep 09 06:47:08.507497 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:08.507033 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerStarted","Data":"c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6"} Sep 09 06:47:08.507497 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:08.507056 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerStarted","Data":"95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183"} Sep 09 06:47:08.534434 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:08.534372 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podStartSLOduration=28.534349478 podStartE2EDuration="28.534349478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:47:08.531752224 +0000 UTC m=+852.856562716" watchObservedRunningTime="2026-09-09 06:47:08.534349478 +0000 UTC m=+852.859159936" Sep 09 06:47:11.311881 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:11.311846 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:11.312401 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:11.311904 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:11.312401 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:11.311924 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:11.312401 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:11.311944 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:11.312401 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:11.311978 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:11.312401 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:11.311998 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:11.313545 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:11.313511 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 09 06:47:11.313671 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:11.313532 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.31:8816: connect: connection refused" Sep 09 06:47:11.313671 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:11.313542 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 09 06:47:14.312625 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:14.312548 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 09 06:47:14.312625 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:14.312550 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.31:8816: connect: connection refused" Sep 09 06:47:14.313198 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:14.312561 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 09 06:47:17.312788 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:17.312754 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:17.313285 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:17.312872 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:17.313450 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:17.313429 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:17.313697 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:17.313680 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:17.332427 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:17.332399 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:47:17.569905 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:47:17.569830 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:48:19.231271 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:19.231240 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x"] Sep 09 06:48:19.234004 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:19.231460 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="online" containerID="cri-o://95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183" gracePeriod=30 Sep 09 06:48:19.234004 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:19.231515 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="offline" containerID="cri-o://c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6" gracePeriod=30 Sep 09 06:48:19.234004 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:19.231529 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="ui" containerID="cri-o://26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667" gracePeriod=30 Sep 09 06:48:19.398729 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:19.398694 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl"] Sep 09 06:48:19.399026 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:19.398995 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="registry" containerID="cri-o://b922b95cdc2e30f6d13d9b45b0bd195f764fb39352adf65576c3e766002be55f" gracePeriod=30 Sep 09 06:48:19.399116 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:19.399088 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="offline" containerID="cri-o://0154628ea18c57355e414a2dd49c1f579b06394754193a0bd038b37a13b1a949" gracePeriod=30 Sep 09 06:48:19.399175 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:19.399085 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="ui" containerID="cri-o://16c63059a002c0ddead680d6247946edb3e5bd615ae60c7425b1399438352a8b" gracePeriod=30 Sep 09 06:48:19.399175 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:19.399112 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="online" containerID="cri-o://5fc6826c5eec4a125b25e2fced03ddae5dd250e0e66da2f866b0d31b7496dd48" gracePeriod=30 Sep 09 06:48:20.820631 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:20.820593 2563 generic.go:356] "Generic (PLEG): container finished" podID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerID="26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667" exitCode=0 Sep 09 06:48:20.820631 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:20.820624 2563 generic.go:356] "Generic (PLEG): container finished" podID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerID="95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183" exitCode=0 Sep 09 06:48:20.821179 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:20.820651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerDied","Data":"26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667"} Sep 09 06:48:20.821179 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:20.820695 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerDied","Data":"95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183"} Sep 09 06:48:20.823718 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:20.823698 2563 generic.go:356] "Generic (PLEG): container finished" podID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerID="16c63059a002c0ddead680d6247946edb3e5bd615ae60c7425b1399438352a8b" exitCode=0 Sep 09 06:48:20.823718 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:20.823716 2563 generic.go:356] "Generic (PLEG): container finished" podID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerID="5fc6826c5eec4a125b25e2fced03ddae5dd250e0e66da2f866b0d31b7496dd48" exitCode=0 Sep 09 06:48:20.823850 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:20.823731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerDied","Data":"16c63059a002c0ddead680d6247946edb3e5bd615ae60c7425b1399438352a8b"} Sep 09 06:48:20.823850 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:20.823757 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerDied","Data":"5fc6826c5eec4a125b25e2fced03ddae5dd250e0e66da2f866b0d31b7496dd48"} Sep 09 06:48:27.314170 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:27.314126 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 09 06:48:27.553209 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:27.553171 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 09 06:48:28.107941 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:28.107897 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 06:48:28.108127 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:28.107908 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 06:48:37.313792 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:37.313746 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 09 06:48:37.553230 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:37.553195 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 09 06:48:38.108182 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:38.108119 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 06:48:38.108371 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:38.108150 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 06:48:47.313443 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:47.313395 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused" Sep 09 06:48:47.313936 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:47.313521 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:48:47.553210 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:47.553165 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 09 06:48:47.553395 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:47.553303 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:48:48.107880 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:48.107828 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Sep 09 06:48:48.108111 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:48.107897 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 09 06:48:48.108111 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:48.107988 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:48:48.108111 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:48.108068 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:48:49.883799 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.883778 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:48:49.937009 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.936986 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-online-tls\") pod \"daff82a0-e8d6-486b-a229-23bc25cb4be8\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " Sep 09 06:48:49.937160 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.937045 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js9pp\" (UniqueName: \"kubernetes.io/projected/daff82a0-e8d6-486b-a229-23bc25cb4be8-kube-api-access-js9pp\") pod \"daff82a0-e8d6-486b-a229-23bc25cb4be8\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " Sep 09 06:48:49.937160 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.937073 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/daff82a0-e8d6-486b-a229-23bc25cb4be8-registry-tls\") pod \"daff82a0-e8d6-486b-a229-23bc25cb4be8\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " Sep 09 06:48:49.937160 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.937122 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-offline-tls\") pod \"daff82a0-e8d6-486b-a229-23bc25cb4be8\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " Sep 09 06:48:49.937160 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.937149 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-ui-tls\") pod \"daff82a0-e8d6-486b-a229-23bc25cb4be8\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " Sep 09 06:48:49.937351 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.937186 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/daff82a0-e8d6-486b-a229-23bc25cb4be8-feast-data\") pod \"daff82a0-e8d6-486b-a229-23bc25cb4be8\" (UID: \"daff82a0-e8d6-486b-a229-23bc25cb4be8\") " Sep 09 06:48:49.937693 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.937649 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daff82a0-e8d6-486b-a229-23bc25cb4be8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "daff82a0-e8d6-486b-a229-23bc25cb4be8" (UID: "daff82a0-e8d6-486b-a229-23bc25cb4be8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 06:48:49.938089 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.938062 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daff82a0-e8d6-486b-a229-23bc25cb4be8-feast-data" (OuterVolumeSpecName: "feast-data") pod "daff82a0-e8d6-486b-a229-23bc25cb4be8" (UID: "daff82a0-e8d6-486b-a229-23bc25cb4be8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:48:49.939425 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.939392 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daff82a0-e8d6-486b-a229-23bc25cb4be8-kube-api-access-js9pp" (OuterVolumeSpecName: "kube-api-access-js9pp") pod "daff82a0-e8d6-486b-a229-23bc25cb4be8" (UID: "daff82a0-e8d6-486b-a229-23bc25cb4be8"). InnerVolumeSpecName "kube-api-access-js9pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:48:49.939425 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.939410 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-online-tls" (OuterVolumeSpecName: "online-tls") pod "daff82a0-e8d6-486b-a229-23bc25cb4be8" (UID: "daff82a0-e8d6-486b-a229-23bc25cb4be8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:48:49.939848 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.939832 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "daff82a0-e8d6-486b-a229-23bc25cb4be8" (UID: "daff82a0-e8d6-486b-a229-23bc25cb4be8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:48:49.939921 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.939906 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "daff82a0-e8d6-486b-a229-23bc25cb4be8" (UID: "daff82a0-e8d6-486b-a229-23bc25cb4be8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:48:49.954638 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.954613 2563 generic.go:356] "Generic (PLEG): container finished" podID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerID="0154628ea18c57355e414a2dd49c1f579b06394754193a0bd038b37a13b1a949" exitCode=137 Sep 09 06:48:49.954638 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.954637 2563 generic.go:356] "Generic (PLEG): container finished" podID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerID="b922b95cdc2e30f6d13d9b45b0bd195f764fb39352adf65576c3e766002be55f" exitCode=137 Sep 09 06:48:49.954817 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.954701 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerDied","Data":"0154628ea18c57355e414a2dd49c1f579b06394754193a0bd038b37a13b1a949"} Sep 09 06:48:49.954817 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.954749 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerDied","Data":"b922b95cdc2e30f6d13d9b45b0bd195f764fb39352adf65576c3e766002be55f"} Sep 09 06:48:49.957407 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.957386 2563 generic.go:356] "Generic (PLEG): container finished" podID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerID="c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6" exitCode=137 Sep 09 06:48:49.957537 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.957450 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" Sep 09 06:48:49.957537 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.957450 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerDied","Data":"c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6"} Sep 09 06:48:49.957657 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.957550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x" event={"ID":"daff82a0-e8d6-486b-a229-23bc25cb4be8","Type":"ContainerDied","Data":"e89c424867a8212159c55cca81c59e8374e82a4dc49288626ef8243f3104f27f"} Sep 09 06:48:49.957657 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.957572 2563 scope.go:117] "RemoveContainer" containerID="26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667" Sep 09 06:48:49.967999 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.967980 2563 scope.go:117] "RemoveContainer" containerID="c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6" Sep 09 06:48:49.977240 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.977226 2563 scope.go:117] "RemoveContainer" containerID="95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183" Sep 09 06:48:49.981063 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.981041 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x"] Sep 09 06:48:49.984743 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.984721 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85bf6597c5-z7l2x"] Sep 09 06:48:49.987335 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.987312 2563 scope.go:117] "RemoveContainer" containerID="2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a" Sep 09 06:48:49.997152 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:49.997117 2563 scope.go:117] "RemoveContainer" containerID="4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780" Sep 09 06:48:50.004987 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.004943 2563 scope.go:117] "RemoveContainer" containerID="26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667" Sep 09 06:48:50.005267 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:48:50.005249 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667\": container with ID starting with 26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667 not found: ID does not exist" containerID="26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667" Sep 09 06:48:50.005311 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.005277 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667"} err="failed to get container status \"26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667\": rpc error: code = NotFound desc = could not find container \"26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667\": container with ID starting with 26dee2f6ed3e99c2149ec8fdec9a53a29124f7a75e65833b6b2ff41e7be78667 not found: ID does not exist" Sep 09 06:48:50.005311 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.005295 2563 scope.go:117] "RemoveContainer" containerID="c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6" Sep 09 06:48:50.005556 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:48:50.005542 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6\": container with ID starting with c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6 not found: ID does not exist" containerID="c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6" Sep 09 06:48:50.005602 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.005559 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6"} err="failed to get container status \"c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6\": rpc error: code = NotFound desc = could not find container \"c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6\": container with ID starting with c94f7b815b8dcef33c0369b1f416d9dd71b30057613b9b0f8d6d9a6322d96ba6 not found: ID does not exist" Sep 09 06:48:50.005602 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.005572 2563 scope.go:117] "RemoveContainer" containerID="95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183" Sep 09 06:48:50.005839 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:48:50.005820 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183\": container with ID starting with 95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183 not found: ID does not exist" containerID="95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183" Sep 09 06:48:50.005896 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.005854 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183"} err="failed to get container status \"95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183\": rpc error: code = NotFound desc = could not find container \"95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183\": container with ID starting with 95ed169dc03725822175bccbdcdf5d8cf375e530a5ade6597ffacccd84d7a183 not found: ID does not exist" Sep 09 06:48:50.005896 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.005869 2563 scope.go:117] "RemoveContainer" containerID="2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a" Sep 09 06:48:50.006138 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:48:50.006122 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a\": container with ID starting with 2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a not found: ID does not exist" containerID="2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a" Sep 09 06:48:50.006181 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.006141 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a"} err="failed to get container status \"2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a\": rpc error: code = NotFound desc = could not find container \"2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a\": container with ID starting with 2a23950b0d4b68833a77371a33c75fd27fc42d2e0544a9a3cfaa0bdb6120ba7a not found: ID does not exist" Sep 09 06:48:50.006181 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.006154 2563 scope.go:117] "RemoveContainer" containerID="4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780" Sep 09 06:48:50.006428 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:48:50.006405 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780\": container with ID starting with 4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780 not found: ID does not exist" containerID="4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780" Sep 09 06:48:50.006547 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.006435 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780"} err="failed to get container status \"4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780\": rpc error: code = NotFound desc = could not find container \"4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780\": container with ID starting with 4a06af31840fcfdb5e92eb9263e54c3af6690b03cd5735f1051cb338c2e72780 not found: ID does not exist" Sep 09 06:48:50.037821 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.037804 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js9pp\" (UniqueName: \"kubernetes.io/projected/daff82a0-e8d6-486b-a229-23bc25cb4be8-kube-api-access-js9pp\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.037920 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.037822 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/daff82a0-e8d6-486b-a229-23bc25cb4be8-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.037920 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.037832 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.037920 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.037839 2563 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-ui-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.037920 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.037846 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/daff82a0-e8d6-486b-a229-23bc25cb4be8-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.037920 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.037854 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/daff82a0-e8d6-486b-a229-23bc25cb4be8-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.043852 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.043834 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:48:50.138515 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.138487 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-ui-tls\") pod \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " Sep 09 06:48:50.138653 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.138528 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-registry-tls\") pod \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " Sep 09 06:48:50.138653 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.138608 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-online-tls\") pod \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " Sep 09 06:48:50.138653 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.138641 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a892fcd7-f82c-4a28-ba48-d79e8b11675c-feast-data\") pod \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " Sep 09 06:48:50.138811 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.138671 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsn4j\" (UniqueName: \"kubernetes.io/projected/a892fcd7-f82c-4a28-ba48-d79e8b11675c-kube-api-access-bsn4j\") pod \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " Sep 09 06:48:50.138811 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.138723 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-offline-tls\") pod \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\" (UID: \"a892fcd7-f82c-4a28-ba48-d79e8b11675c\") " Sep 09 06:48:50.139280 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.139247 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a892fcd7-f82c-4a28-ba48-d79e8b11675c-feast-data" (OuterVolumeSpecName: "feast-data") pod "a892fcd7-f82c-4a28-ba48-d79e8b11675c" (UID: "a892fcd7-f82c-4a28-ba48-d79e8b11675c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:48:50.140926 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.140894 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a892fcd7-f82c-4a28-ba48-d79e8b11675c" (UID: "a892fcd7-f82c-4a28-ba48-d79e8b11675c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:48:50.141084 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.140952 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a892fcd7-f82c-4a28-ba48-d79e8b11675c" (UID: "a892fcd7-f82c-4a28-ba48-d79e8b11675c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:48:50.141084 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.140999 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a892fcd7-f82c-4a28-ba48-d79e8b11675c-kube-api-access-bsn4j" (OuterVolumeSpecName: "kube-api-access-bsn4j") pod "a892fcd7-f82c-4a28-ba48-d79e8b11675c" (UID: "a892fcd7-f82c-4a28-ba48-d79e8b11675c"). InnerVolumeSpecName "kube-api-access-bsn4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:48:50.141201 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.141182 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-online-tls" (OuterVolumeSpecName: "online-tls") pod "a892fcd7-f82c-4a28-ba48-d79e8b11675c" (UID: "a892fcd7-f82c-4a28-ba48-d79e8b11675c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:48:50.141356 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.141334 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a892fcd7-f82c-4a28-ba48-d79e8b11675c" (UID: "a892fcd7-f82c-4a28-ba48-d79e8b11675c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:48:50.147359 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.147336 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" path="/var/lib/kubelet/pods/daff82a0-e8d6-486b-a229-23bc25cb4be8/volumes" Sep 09 06:48:50.239335 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.239310 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a892fcd7-f82c-4a28-ba48-d79e8b11675c-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.239335 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.239332 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsn4j\" (UniqueName: \"kubernetes.io/projected/a892fcd7-f82c-4a28-ba48-d79e8b11675c-kube-api-access-bsn4j\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.239492 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.239342 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.239492 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.239351 2563 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-ui-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.239492 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.239358 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.239492 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.239365 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a892fcd7-f82c-4a28-ba48-d79e8b11675c-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:48:50.965615 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.965573 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" event={"ID":"a892fcd7-f82c-4a28-ba48-d79e8b11675c","Type":"ContainerDied","Data":"a479a6271e293b0c34667a3ed5071f0d9477601007c4fe4a325c6828f7f84a0d"} Sep 09 06:48:50.966137 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.965642 2563 scope.go:117] "RemoveContainer" containerID="16c63059a002c0ddead680d6247946edb3e5bd615ae60c7425b1399438352a8b" Sep 09 06:48:50.966137 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.965658 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl" Sep 09 06:48:50.974756 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.974735 2563 scope.go:117] "RemoveContainer" containerID="0154628ea18c57355e414a2dd49c1f579b06394754193a0bd038b37a13b1a949" Sep 09 06:48:50.982926 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.982907 2563 scope.go:117] "RemoveContainer" containerID="5fc6826c5eec4a125b25e2fced03ddae5dd250e0e66da2f866b0d31b7496dd48" Sep 09 06:48:50.988500 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.988478 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl"] Sep 09 06:48:50.991821 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.991798 2563 scope.go:117] "RemoveContainer" containerID="b922b95cdc2e30f6d13d9b45b0bd195f764fb39352adf65576c3e766002be55f" Sep 09 06:48:50.992870 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.992771 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b94879c9f-gdfhl"] Sep 09 06:48:50.999707 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:50.999691 2563 scope.go:117] "RemoveContainer" containerID="f49c50c406cf66b90e86772829b09e3c08694b39c8e8b16b417e3ea4a628ad09" Sep 09 06:48:51.006800 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:51.006777 2563 scope.go:117] "RemoveContainer" containerID="ee864b6837037117170d2b711ba02b19dfec6bae934b6c1fc022c0ac9e6e6504" Sep 09 06:48:52.146773 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:48:52.146738 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" path="/var/lib/kubelet/pods/a892fcd7-f82c-4a28-ba48-d79e8b11675c/volumes" Sep 09 06:49:02.175398 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175363 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-2bsbw"] Sep 09 06:49:02.175841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175792 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="feast-init" Sep 09 06:49:02.175841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175810 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="feast-init" Sep 09 06:49:02.175841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175822 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="offline" Sep 09 06:49:02.175841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175829 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="offline" Sep 09 06:49:02.175841 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175840 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="feast-apply" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175848 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="feast-apply" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175858 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="feast-apply" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175866 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="feast-apply" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175875 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="online" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175885 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="online" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175903 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="ui" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175910 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="ui" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175918 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="offline" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175926 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="offline" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175935 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="ui" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175943 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="ui" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175987 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="feast-init" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.175996 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="feast-init" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176010 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="registry" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176018 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="registry" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176026 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="online" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176034 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="online" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176115 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="offline" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176127 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="registry" Sep 09 06:49:02.176126 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176135 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="online" Sep 09 06:49:02.177051 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176143 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="daff82a0-e8d6-486b-a229-23bc25cb4be8" containerName="ui" Sep 09 06:49:02.177051 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176152 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="offline" Sep 09 06:49:02.177051 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176163 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="ui" Sep 09 06:49:02.177051 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.176170 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a892fcd7-f82c-4a28-ba48-d79e8b11675c" containerName="online" Sep 09 06:49:02.180754 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.180734 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:49:02.183469 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.183448 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-gjh4p\"" Sep 09 06:49:02.183570 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.183491 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 06:49:02.184460 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.184446 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 09 06:49:02.184547 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.184486 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 06:49:02.196199 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.190902 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-2bsbw"] Sep 09 06:49:02.205292 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.205265 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-4h59x"] Sep 09 06:49:02.208435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.208415 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4h59x" Sep 09 06:49:02.217378 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.217353 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4h59x"] Sep 09 06:49:02.320649 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.320607 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d14ed526-5359-490c-b5b8-aae753fc64a9-postgresdata\") pod \"postgres-864846d4f5-2bsbw\" (UID: \"d14ed526-5359-490c-b5b8-aae753fc64a9\") " pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:49:02.320801 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.320669 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk7bp\" (UniqueName: \"kubernetes.io/projected/93ec4a3f-d056-4caf-bd0a-0959550002e0-kube-api-access-tk7bp\") pod \"redis-69bbd798b6-4h59x\" (UID: \"93ec4a3f-d056-4caf-bd0a-0959550002e0\") " pod="test-ns-feast/redis-69bbd798b6-4h59x" Sep 09 06:49:02.320801 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.320733 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lltz\" (UniqueName: \"kubernetes.io/projected/d14ed526-5359-490c-b5b8-aae753fc64a9-kube-api-access-7lltz\") pod \"postgres-864846d4f5-2bsbw\" (UID: \"d14ed526-5359-490c-b5b8-aae753fc64a9\") " pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:49:02.421797 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.421767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lltz\" (UniqueName: \"kubernetes.io/projected/d14ed526-5359-490c-b5b8-aae753fc64a9-kube-api-access-7lltz\") pod \"postgres-864846d4f5-2bsbw\" (UID: \"d14ed526-5359-490c-b5b8-aae753fc64a9\") " pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:49:02.421950 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.421816 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d14ed526-5359-490c-b5b8-aae753fc64a9-postgresdata\") pod \"postgres-864846d4f5-2bsbw\" (UID: \"d14ed526-5359-490c-b5b8-aae753fc64a9\") " pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:49:02.421950 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.421857 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk7bp\" (UniqueName: \"kubernetes.io/projected/93ec4a3f-d056-4caf-bd0a-0959550002e0-kube-api-access-tk7bp\") pod \"redis-69bbd798b6-4h59x\" (UID: \"93ec4a3f-d056-4caf-bd0a-0959550002e0\") " pod="test-ns-feast/redis-69bbd798b6-4h59x" Sep 09 06:49:02.422261 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.422241 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d14ed526-5359-490c-b5b8-aae753fc64a9-postgresdata\") pod \"postgres-864846d4f5-2bsbw\" (UID: \"d14ed526-5359-490c-b5b8-aae753fc64a9\") " pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:49:02.430813 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.430754 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lltz\" (UniqueName: \"kubernetes.io/projected/d14ed526-5359-490c-b5b8-aae753fc64a9-kube-api-access-7lltz\") pod \"postgres-864846d4f5-2bsbw\" (UID: \"d14ed526-5359-490c-b5b8-aae753fc64a9\") " pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:49:02.430813 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.430756 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk7bp\" (UniqueName: \"kubernetes.io/projected/93ec4a3f-d056-4caf-bd0a-0959550002e0-kube-api-access-tk7bp\") pod \"redis-69bbd798b6-4h59x\" (UID: \"93ec4a3f-d056-4caf-bd0a-0959550002e0\") " pod="test-ns-feast/redis-69bbd798b6-4h59x" Sep 09 06:49:02.502331 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.502309 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:49:02.519266 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.519244 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4h59x" Sep 09 06:49:02.780522 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.780405 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-2bsbw"] Sep 09 06:49:02.782616 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:49:02.782589 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd14ed526_5359_490c_b5b8_aae753fc64a9.slice/crio-0663eeff8436e303933ae13b5e841e86f6ec82f63923bed304daf035fc8e768b WatchSource:0}: Error finding container 0663eeff8436e303933ae13b5e841e86f6ec82f63923bed304daf035fc8e768b: Status 404 returned error can't find the container with id 0663eeff8436e303933ae13b5e841e86f6ec82f63923bed304daf035fc8e768b Sep 09 06:49:02.784540 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.784520 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 06:49:02.803436 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:02.803416 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4h59x"] Sep 09 06:49:02.805671 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:49:02.805639 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93ec4a3f_d056_4caf_bd0a_0959550002e0.slice/crio-da011231a60e6b345e12704264dea27498ff71674ba1dfedd26010fa6cfce2a7 WatchSource:0}: Error finding container da011231a60e6b345e12704264dea27498ff71674ba1dfedd26010fa6cfce2a7: Status 404 returned error can't find the container with id da011231a60e6b345e12704264dea27498ff71674ba1dfedd26010fa6cfce2a7 Sep 09 06:49:03.017489 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:03.017451 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4h59x" event={"ID":"93ec4a3f-d056-4caf-bd0a-0959550002e0","Type":"ContainerStarted","Data":"da011231a60e6b345e12704264dea27498ff71674ba1dfedd26010fa6cfce2a7"} Sep 09 06:49:03.018546 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:03.018528 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-2bsbw" event={"ID":"d14ed526-5359-490c-b5b8-aae753fc64a9","Type":"ContainerStarted","Data":"0663eeff8436e303933ae13b5e841e86f6ec82f63923bed304daf035fc8e768b"} Sep 09 06:49:05.029456 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:05.029364 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4h59x" event={"ID":"93ec4a3f-d056-4caf-bd0a-0959550002e0","Type":"ContainerStarted","Data":"301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9"} Sep 09 06:49:05.044280 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:05.044238 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-4h59x" podStartSLOduration=1.24391115 podStartE2EDuration="3.044224136s" totalImagesPullingTime="1.800312986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:49:05.043792807 +0000 UTC m=+969.368603263" watchObservedRunningTime="2026-09-09 06:49:05.044224136 +0000 UTC m=+969.369034592" Sep 09 06:49:08.044207 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:08.044171 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-2bsbw" event={"ID":"d14ed526-5359-490c-b5b8-aae753fc64a9","Type":"ContainerStarted","Data":"c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb"} Sep 09 06:49:08.059772 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:08.059728 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-2bsbw" podStartSLOduration=1.657238799 podStartE2EDuration="6.059707226s" totalImagesPullingTime="4.402468427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:49:08.059267157 +0000 UTC m=+972.384077614" watchObservedRunningTime="2026-09-09 06:49:08.059707226 +0000 UTC m=+972.384517681" Sep 09 06:49:10.996662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:10.996624 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq"] Sep 09 06:49:11.000461 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.000445 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.002887 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.002859 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 09 06:49:11.003024 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.003006 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 09 06:49:11.003348 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.003334 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 09 06:49:11.003403 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.003361 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-88x2v\"" Sep 09 06:49:11.004216 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.004183 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 09 06:49:11.011799 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.011777 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq"] Sep 09 06:49:11.089252 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.089225 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2f9g\" (UniqueName: \"kubernetes.io/projected/a6294c35-53d6-4990-944a-1f7e0cc51f9b-kube-api-access-h2f9g\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.089377 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.089264 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-online-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.089377 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.089338 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-registry-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.089377 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.089361 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-offline-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.089479 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.089386 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6294c35-53d6-4990-944a-1f7e0cc51f9b-feast-data\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.190229 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.190206 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-registry-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.190328 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.190235 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-offline-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.190328 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.190250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6294c35-53d6-4990-944a-1f7e0cc51f9b-feast-data\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.190328 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.190281 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f9g\" (UniqueName: \"kubernetes.io/projected/a6294c35-53d6-4990-944a-1f7e0cc51f9b-kube-api-access-h2f9g\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.190328 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.190320 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-online-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.190460 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:49:11.190365 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 09 06:49:11.190460 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:49:11.190373 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Sep 09 06:49:11.190460 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:49:11.190414 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 09 06:49:11.190460 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:49:11.190437 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-offline-tls podName:a6294c35-53d6-4990-944a-1f7e0cc51f9b nodeName:}" failed. No retries permitted until 2026-09-09 06:49:11.690417583 +0000 UTC m=+976.015228018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-offline-tls") pod "feast-credit-scoring-77b76d646b-6qpdq" (UID: "a6294c35-53d6-4990-944a-1f7e0cc51f9b") : secret "feast-credit-scoring-offline-tls" not found Sep 09 06:49:11.190460 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:49:11.190453 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-registry-tls podName:a6294c35-53d6-4990-944a-1f7e0cc51f9b nodeName:}" failed. No retries permitted until 2026-09-09 06:49:11.690444394 +0000 UTC m=+976.015254828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-registry-tls") pod "feast-credit-scoring-77b76d646b-6qpdq" (UID: "a6294c35-53d6-4990-944a-1f7e0cc51f9b") : secret "feast-credit-scoring-registry-tls" not found Sep 09 06:49:11.190616 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:49:11.190464 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-online-tls podName:a6294c35-53d6-4990-944a-1f7e0cc51f9b nodeName:}" failed. No retries permitted until 2026-09-09 06:49:11.690458701 +0000 UTC m=+976.015269136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-online-tls") pod "feast-credit-scoring-77b76d646b-6qpdq" (UID: "a6294c35-53d6-4990-944a-1f7e0cc51f9b") : secret "feast-credit-scoring-online-tls" not found Sep 09 06:49:11.190649 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.190614 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6294c35-53d6-4990-944a-1f7e0cc51f9b-feast-data\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.200704 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.200685 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2f9g\" (UniqueName: \"kubernetes.io/projected/a6294c35-53d6-4990-944a-1f7e0cc51f9b-kube-api-access-h2f9g\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.694663 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.694634 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-online-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.694859 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.694685 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-registry-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.694859 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.694715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-offline-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.697428 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.697398 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-offline-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.697532 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.697398 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-online-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.697532 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.697454 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-registry-tls\") pod \"feast-credit-scoring-77b76d646b-6qpdq\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:11.910065 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:11.910032 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:12.193282 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:12.193258 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq"] Sep 09 06:49:12.195604 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:49:12.195578 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6294c35_53d6_4990_944a_1f7e0cc51f9b.slice/crio-440f541fe94c90d6e970cf0fc5a6ffe3833e30e54bf9a4927336cbf30d397c5b WatchSource:0}: Error finding container 440f541fe94c90d6e970cf0fc5a6ffe3833e30e54bf9a4927336cbf30d397c5b: Status 404 returned error can't find the container with id 440f541fe94c90d6e970cf0fc5a6ffe3833e30e54bf9a4927336cbf30d397c5b Sep 09 06:49:13.065982 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:13.065922 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerStarted","Data":"4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3"} Sep 09 06:49:13.066150 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:13.066044 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerStarted","Data":"440f541fe94c90d6e970cf0fc5a6ffe3833e30e54bf9a4927336cbf30d397c5b"} Sep 09 06:49:14.071621 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:14.071587 2563 generic.go:356] "Generic (PLEG): container finished" podID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerID="4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3" exitCode=0 Sep 09 06:49:14.071947 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:14.071674 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerDied","Data":"4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3"} Sep 09 06:49:15.077625 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:15.077590 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerStarted","Data":"86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786"} Sep 09 06:49:21.105427 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:21.105387 2563 generic.go:356] "Generic (PLEG): container finished" podID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerID="86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786" exitCode=0 Sep 09 06:49:21.105828 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:21.105464 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerDied","Data":"86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786"} Sep 09 06:49:22.114588 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:22.114556 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerStarted","Data":"457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672"} Sep 09 06:49:22.115080 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:22.114622 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerStarted","Data":"3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b"} Sep 09 06:49:22.115080 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:22.114646 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerStarted","Data":"6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226"} Sep 09 06:49:22.143561 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:22.143499 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podStartSLOduration=12.14347721 podStartE2EDuration="12.14347721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:49:22.14033613 +0000 UTC m=+986.465146584" watchObservedRunningTime="2026-09-09 06:49:22.14347721 +0000 UTC m=+986.468287666" Sep 09 06:49:23.910508 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:23.910470 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:23.910896 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:23.910567 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:23.911372 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:23.911351 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:23.912489 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:23.912449 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" Sep 09 06:49:23.912631 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:23.912484 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.34:8816: connect: connection refused" Sep 09 06:49:23.912631 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:23.912519 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 09 06:49:26.911105 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:26.911057 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.34:8816: connect: connection refused" Sep 09 06:49:26.911567 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:26.911112 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 09 06:49:26.911567 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:26.911200 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" Sep 09 06:49:29.910567 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:29.910529 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" Sep 09 06:49:29.911514 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:29.911494 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:29.911946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:29.911927 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:29.912383 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:29.912366 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:29.919026 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:29.919007 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:30.156946 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:30.156918 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:30.160266 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:30.160240 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:31.910643 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:31.910593 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:32.910711 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:32.910669 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" Sep 09 06:49:35.911123 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:35.911088 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:49:35.911880 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:49:35.911861 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:50:39.900736 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:39.900694 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq"] Sep 09 06:50:39.901382 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:39.900976 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="registry" containerID="cri-o://6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226" gracePeriod=30 Sep 09 06:50:39.901382 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:39.901029 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="offline" containerID="cri-o://457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672" gracePeriod=30 Sep 09 06:50:39.901382 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:39.901039 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" containerID="cri-o://3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b" gracePeriod=30 Sep 09 06:50:39.916270 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:39.916237 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-2bsbw"] Sep 09 06:50:39.916440 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:39.916419 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-2bsbw" podUID="d14ed526-5359-490c-b5b8-aae753fc64a9" containerName="postgres" containerID="cri-o://c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb" gracePeriod=30 Sep 09 06:50:39.921695 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:39.921657 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4h59x"] Sep 09 06:50:39.921892 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:39.921841 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-4h59x" podUID="93ec4a3f-d056-4caf-bd0a-0959550002e0" containerName="redis" containerID="cri-o://301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9" gracePeriod=30 Sep 09 06:50:40.157604 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.157528 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 09 06:50:40.183210 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.183189 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:50:40.186393 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.186377 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4h59x" Sep 09 06:50:40.310303 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.310275 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lltz\" (UniqueName: \"kubernetes.io/projected/d14ed526-5359-490c-b5b8-aae753fc64a9-kube-api-access-7lltz\") pod \"d14ed526-5359-490c-b5b8-aae753fc64a9\" (UID: \"d14ed526-5359-490c-b5b8-aae753fc64a9\") " Sep 09 06:50:40.310472 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.310348 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk7bp\" (UniqueName: \"kubernetes.io/projected/93ec4a3f-d056-4caf-bd0a-0959550002e0-kube-api-access-tk7bp\") pod \"93ec4a3f-d056-4caf-bd0a-0959550002e0\" (UID: \"93ec4a3f-d056-4caf-bd0a-0959550002e0\") " Sep 09 06:50:40.310472 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.310368 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d14ed526-5359-490c-b5b8-aae753fc64a9-postgresdata\") pod \"d14ed526-5359-490c-b5b8-aae753fc64a9\" (UID: \"d14ed526-5359-490c-b5b8-aae753fc64a9\") " Sep 09 06:50:40.312734 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.312691 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ec4a3f-d056-4caf-bd0a-0959550002e0-kube-api-access-tk7bp" (OuterVolumeSpecName: "kube-api-access-tk7bp") pod "93ec4a3f-d056-4caf-bd0a-0959550002e0" (UID: "93ec4a3f-d056-4caf-bd0a-0959550002e0"). InnerVolumeSpecName "kube-api-access-tk7bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:50:40.312877 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.312855 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14ed526-5359-490c-b5b8-aae753fc64a9-kube-api-access-7lltz" (OuterVolumeSpecName: "kube-api-access-7lltz") pod "d14ed526-5359-490c-b5b8-aae753fc64a9" (UID: "d14ed526-5359-490c-b5b8-aae753fc64a9"). InnerVolumeSpecName "kube-api-access-7lltz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:50:40.340265 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.340232 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14ed526-5359-490c-b5b8-aae753fc64a9-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "d14ed526-5359-490c-b5b8-aae753fc64a9" (UID: "d14ed526-5359-490c-b5b8-aae753fc64a9"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:50:40.411809 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.411723 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lltz\" (UniqueName: \"kubernetes.io/projected/d14ed526-5359-490c-b5b8-aae753fc64a9-kube-api-access-7lltz\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:50:40.411809 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.411751 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk7bp\" (UniqueName: \"kubernetes.io/projected/93ec4a3f-d056-4caf-bd0a-0959550002e0-kube-api-access-tk7bp\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:50:40.411809 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.411761 2563 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d14ed526-5359-490c-b5b8-aae753fc64a9-postgresdata\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:50:40.462449 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.462410 2563 generic.go:356] "Generic (PLEG): container finished" podID="93ec4a3f-d056-4caf-bd0a-0959550002e0" containerID="301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9" exitCode=0 Sep 09 06:50:40.462449 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.462453 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4h59x" Sep 09 06:50:40.462677 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.462510 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4h59x" event={"ID":"93ec4a3f-d056-4caf-bd0a-0959550002e0","Type":"ContainerDied","Data":"301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9"} Sep 09 06:50:40.462677 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.462561 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4h59x" event={"ID":"93ec4a3f-d056-4caf-bd0a-0959550002e0","Type":"ContainerDied","Data":"da011231a60e6b345e12704264dea27498ff71674ba1dfedd26010fa6cfce2a7"} Sep 09 06:50:40.462677 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.462582 2563 scope.go:117] "RemoveContainer" containerID="301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9" Sep 09 06:50:40.464036 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.464013 2563 generic.go:356] "Generic (PLEG): container finished" podID="d14ed526-5359-490c-b5b8-aae753fc64a9" containerID="c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb" exitCode=0 Sep 09 06:50:40.464239 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.464045 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-2bsbw" event={"ID":"d14ed526-5359-490c-b5b8-aae753fc64a9","Type":"ContainerDied","Data":"c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb"} Sep 09 06:50:40.464374 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.464260 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-2bsbw" event={"ID":"d14ed526-5359-490c-b5b8-aae753fc64a9","Type":"ContainerDied","Data":"0663eeff8436e303933ae13b5e841e86f6ec82f63923bed304daf035fc8e768b"} Sep 09 06:50:40.464374 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.464062 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-2bsbw" Sep 09 06:50:40.472768 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.472709 2563 scope.go:117] "RemoveContainer" containerID="301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9" Sep 09 06:50:40.473070 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:50:40.473046 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9\": container with ID starting with 301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9 not found: ID does not exist" containerID="301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9" Sep 09 06:50:40.473147 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.473084 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9"} err="failed to get container status \"301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9\": rpc error: code = NotFound desc = could not find container \"301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9\": container with ID starting with 301890aace45a920adc99c957ef19fb58c3d324fe81f66c3fd936eb789b20bf9 not found: ID does not exist" Sep 09 06:50:40.473147 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.473107 2563 scope.go:117] "RemoveContainer" containerID="c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb" Sep 09 06:50:40.480671 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.480656 2563 scope.go:117] "RemoveContainer" containerID="c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb" Sep 09 06:50:40.480939 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:50:40.480922 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb\": container with ID starting with c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb not found: ID does not exist" containerID="c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb" Sep 09 06:50:40.481035 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.480943 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb"} err="failed to get container status \"c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb\": rpc error: code = NotFound desc = could not find container \"c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb\": container with ID starting with c42f69c108dcdbd6cc3ec67c0681e5328d14dba23fe4ab895520eb4a91be11fb not found: ID does not exist" Sep 09 06:50:40.487110 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.487089 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-2bsbw"] Sep 09 06:50:40.489615 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.489596 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-2bsbw"] Sep 09 06:50:40.499821 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.499796 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4h59x"] Sep 09 06:50:40.501336 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:40.501317 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4h59x"] Sep 09 06:50:41.471451 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:41.471415 2563 generic.go:356] "Generic (PLEG): container finished" podID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerID="3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b" exitCode=0 Sep 09 06:50:41.471842 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:41.471491 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerDied","Data":"3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b"} Sep 09 06:50:42.147315 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:42.147279 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93ec4a3f-d056-4caf-bd0a-0959550002e0" path="/var/lib/kubelet/pods/93ec4a3f-d056-4caf-bd0a-0959550002e0/volumes" Sep 09 06:50:42.147613 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:42.147600 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d14ed526-5359-490c-b5b8-aae753fc64a9" path="/var/lib/kubelet/pods/d14ed526-5359-490c-b5b8-aae753fc64a9/volumes" Sep 09 06:50:50.157221 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:50:50.157184 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 09 06:51:00.157290 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:00.157249 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 09 06:51:00.157665 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:00.157359 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:51:10.157306 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.157229 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 09 06:51:10.546228 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.546208 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:51:10.600413 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.600384 2563 generic.go:356] "Generic (PLEG): container finished" podID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerID="457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672" exitCode=137 Sep 09 06:51:10.600413 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.600407 2563 generic.go:356] "Generic (PLEG): container finished" podID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerID="6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226" exitCode=137 Sep 09 06:51:10.600589 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.600440 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" Sep 09 06:51:10.600589 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.600447 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerDied","Data":"457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672"} Sep 09 06:51:10.600589 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.600520 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerDied","Data":"6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226"} Sep 09 06:51:10.600589 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.600548 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq" event={"ID":"a6294c35-53d6-4990-944a-1f7e0cc51f9b","Type":"ContainerDied","Data":"440f541fe94c90d6e970cf0fc5a6ffe3833e30e54bf9a4927336cbf30d397c5b"} Sep 09 06:51:10.600589 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.600583 2563 scope.go:117] "RemoveContainer" containerID="457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672" Sep 09 06:51:10.608652 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.608636 2563 scope.go:117] "RemoveContainer" containerID="3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b" Sep 09 06:51:10.614730 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.614706 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-offline-tls\") pod \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " Sep 09 06:51:10.614826 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.614755 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2f9g\" (UniqueName: \"kubernetes.io/projected/a6294c35-53d6-4990-944a-1f7e0cc51f9b-kube-api-access-h2f9g\") pod \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " Sep 09 06:51:10.614826 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.614793 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-online-tls\") pod \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " Sep 09 06:51:10.614925 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.614840 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6294c35-53d6-4990-944a-1f7e0cc51f9b-feast-data\") pod \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " Sep 09 06:51:10.615079 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.614990 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-registry-tls\") pod \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\" (UID: \"a6294c35-53d6-4990-944a-1f7e0cc51f9b\") " Sep 09 06:51:10.616497 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.616471 2563 scope.go:117] "RemoveContainer" containerID="6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226" Sep 09 06:51:10.617074 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.617050 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-online-tls" (OuterVolumeSpecName: "online-tls") pod "a6294c35-53d6-4990-944a-1f7e0cc51f9b" (UID: "a6294c35-53d6-4990-944a-1f7e0cc51f9b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:51:10.617257 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.617226 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a6294c35-53d6-4990-944a-1f7e0cc51f9b" (UID: "a6294c35-53d6-4990-944a-1f7e0cc51f9b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:51:10.617490 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.617461 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a6294c35-53d6-4990-944a-1f7e0cc51f9b" (UID: "a6294c35-53d6-4990-944a-1f7e0cc51f9b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:51:10.617569 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.617514 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6294c35-53d6-4990-944a-1f7e0cc51f9b-kube-api-access-h2f9g" (OuterVolumeSpecName: "kube-api-access-h2f9g") pod "a6294c35-53d6-4990-944a-1f7e0cc51f9b" (UID: "a6294c35-53d6-4990-944a-1f7e0cc51f9b"). InnerVolumeSpecName "kube-api-access-h2f9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:51:10.624317 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.624287 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6294c35-53d6-4990-944a-1f7e0cc51f9b-feast-data" (OuterVolumeSpecName: "feast-data") pod "a6294c35-53d6-4990-944a-1f7e0cc51f9b" (UID: "a6294c35-53d6-4990-944a-1f7e0cc51f9b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:51:10.630107 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.630090 2563 scope.go:117] "RemoveContainer" containerID="86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786" Sep 09 06:51:10.637048 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.637030 2563 scope.go:117] "RemoveContainer" containerID="4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3" Sep 09 06:51:10.643652 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.643635 2563 scope.go:117] "RemoveContainer" containerID="457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672" Sep 09 06:51:10.643924 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:51:10.643896 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672\": container with ID starting with 457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672 not found: ID does not exist" containerID="457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672" Sep 09 06:51:10.644011 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.643935 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672"} err="failed to get container status \"457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672\": rpc error: code = NotFound desc = could not find container \"457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672\": container with ID starting with 457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672 not found: ID does not exist" Sep 09 06:51:10.644011 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.644002 2563 scope.go:117] "RemoveContainer" containerID="3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b" Sep 09 06:51:10.644268 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:51:10.644251 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b\": container with ID starting with 3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b not found: ID does not exist" containerID="3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b" Sep 09 06:51:10.644327 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.644273 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b"} err="failed to get container status \"3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b\": rpc error: code = NotFound desc = could not find container \"3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b\": container with ID starting with 3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b not found: ID does not exist" Sep 09 06:51:10.644327 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.644289 2563 scope.go:117] "RemoveContainer" containerID="6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226" Sep 09 06:51:10.644563 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:51:10.644546 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226\": container with ID starting with 6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226 not found: ID does not exist" containerID="6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226" Sep 09 06:51:10.644601 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.644567 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226"} err="failed to get container status \"6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226\": rpc error: code = NotFound desc = could not find container \"6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226\": container with ID starting with 6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226 not found: ID does not exist" Sep 09 06:51:10.644601 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.644582 2563 scope.go:117] "RemoveContainer" containerID="86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786" Sep 09 06:51:10.644799 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:51:10.644784 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786\": container with ID starting with 86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786 not found: ID does not exist" containerID="86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786" Sep 09 06:51:10.644834 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.644805 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786"} err="failed to get container status \"86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786\": rpc error: code = NotFound desc = could not find container \"86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786\": container with ID starting with 86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786 not found: ID does not exist" Sep 09 06:51:10.644834 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.644819 2563 scope.go:117] "RemoveContainer" containerID="4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3" Sep 09 06:51:10.645063 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:51:10.645043 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3\": container with ID starting with 4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3 not found: ID does not exist" containerID="4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3" Sep 09 06:51:10.645120 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.645068 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3"} err="failed to get container status \"4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3\": rpc error: code = NotFound desc = could not find container \"4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3\": container with ID starting with 4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3 not found: ID does not exist" Sep 09 06:51:10.645120 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.645081 2563 scope.go:117] "RemoveContainer" containerID="457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672" Sep 09 06:51:10.645317 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.645299 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672"} err="failed to get container status \"457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672\": rpc error: code = NotFound desc = could not find container \"457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672\": container with ID starting with 457fd2b3681405fc907c178fa5311e9c7a71979f6b762dae25895cc25e0a3672 not found: ID does not exist" Sep 09 06:51:10.645357 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.645318 2563 scope.go:117] "RemoveContainer" containerID="3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b" Sep 09 06:51:10.645559 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.645541 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b"} err="failed to get container status \"3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b\": rpc error: code = NotFound desc = could not find container \"3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b\": container with ID starting with 3d75da504c9258b548e1e98272f7ae78549035383cafedced46466b1e8f0046b not found: ID does not exist" Sep 09 06:51:10.645601 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.645560 2563 scope.go:117] "RemoveContainer" containerID="6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226" Sep 09 06:51:10.645779 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.645763 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226"} err="failed to get container status \"6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226\": rpc error: code = NotFound desc = could not find container \"6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226\": container with ID starting with 6c5d8aa3f576738bdd98e55390a710eee9b1734ac4ce81e24a336b50a5e32226 not found: ID does not exist" Sep 09 06:51:10.645826 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.645779 2563 scope.go:117] "RemoveContainer" containerID="86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786" Sep 09 06:51:10.646014 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.645991 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786"} err="failed to get container status \"86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786\": rpc error: code = NotFound desc = could not find container \"86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786\": container with ID starting with 86d96a4cfa50be1edbda35704dcb0de75f7e4cb02f0dad3710863bf1214d9786 not found: ID does not exist" Sep 09 06:51:10.646057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.646016 2563 scope.go:117] "RemoveContainer" containerID="4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3" Sep 09 06:51:10.646275 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.646257 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3"} err="failed to get container status \"4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3\": rpc error: code = NotFound desc = could not find container \"4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3\": container with ID starting with 4f01d94c7cc9efbf70f8587ce0cbc714d4e9766e31a1afd55a2d570e6d4041f3 not found: ID does not exist" Sep 09 06:51:10.716613 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.716567 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2f9g\" (UniqueName: \"kubernetes.io/projected/a6294c35-53d6-4990-944a-1f7e0cc51f9b-kube-api-access-h2f9g\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:51:10.716613 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.716587 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:51:10.716613 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.716596 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6294c35-53d6-4990-944a-1f7e0cc51f9b-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:51:10.716613 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.716603 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:51:10.716613 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.716611 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6294c35-53d6-4990-944a-1f7e0cc51f9b-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:51:10.923715 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.923689 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq"] Sep 09 06:51:10.927231 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:10.927210 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-77b76d646b-6qpdq"] Sep 09 06:51:12.147762 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:51:12.147732 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" path="/var/lib/kubelet/pods/a6294c35-53d6-4990-944a-1f7e0cc51f9b/volumes" Sep 09 06:53:16.981436 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981392 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-mh9l8"] Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981708 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d14ed526-5359-490c-b5b8-aae753fc64a9" containerName="postgres" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981721 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d14ed526-5359-490c-b5b8-aae753fc64a9" containerName="postgres" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981738 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="feast-init" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981743 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="feast-init" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981751 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="offline" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981757 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="offline" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981768 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ec4a3f-d056-4caf-bd0a-0959550002e0" containerName="redis" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981773 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ec4a3f-d056-4caf-bd0a-0959550002e0" containerName="redis" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981779 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="feast-apply" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981784 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="feast-apply" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981790 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="registry" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981797 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="registry" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981811 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981819 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981869 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="online" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981877 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d14ed526-5359-490c-b5b8-aae753fc64a9" containerName="postgres" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981884 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ec4a3f-d056-4caf-bd0a-0959550002e0" containerName="redis" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981892 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="offline" Sep 09 06:53:16.982050 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.981902 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6294c35-53d6-4990-944a-1f7e0cc51f9b" containerName="registry" Sep 09 06:53:16.984883 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.984866 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" Sep 09 06:53:16.987940 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.987922 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-8gdzp\"" Sep 09 06:53:16.988227 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.988212 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 09 06:53:16.989105 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:16.989084 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 09 06:53:17.009920 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:17.009895 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-mh9l8"] Sep 09 06:53:17.123537 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:17.123511 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c97kl\" (UniqueName: \"kubernetes.io/projected/628c588a-3cda-46e3-8532-8d5893ac1ef0-kube-api-access-c97kl\") pod \"redis-69bbd798b6-mh9l8\" (UID: \"628c588a-3cda-46e3-8532-8d5893ac1ef0\") " pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" Sep 09 06:53:17.224400 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:17.224374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c97kl\" (UniqueName: \"kubernetes.io/projected/628c588a-3cda-46e3-8532-8d5893ac1ef0-kube-api-access-c97kl\") pod \"redis-69bbd798b6-mh9l8\" (UID: \"628c588a-3cda-46e3-8532-8d5893ac1ef0\") " pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" Sep 09 06:53:17.233606 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:17.233542 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c97kl\" (UniqueName: \"kubernetes.io/projected/628c588a-3cda-46e3-8532-8d5893ac1ef0-kube-api-access-c97kl\") pod \"redis-69bbd798b6-mh9l8\" (UID: \"628c588a-3cda-46e3-8532-8d5893ac1ef0\") " pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" Sep 09 06:53:17.306128 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:17.306102 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" Sep 09 06:53:17.574279 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:17.574256 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-mh9l8"] Sep 09 06:53:17.576762 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:53:17.576736 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod628c588a_3cda_46e3_8532_8d5893ac1ef0.slice/crio-8827fbe2975e80869c1a989eb6666bba4965efd644436f8446314dbb5436cec7 WatchSource:0}: Error finding container 8827fbe2975e80869c1a989eb6666bba4965efd644436f8446314dbb5436cec7: Status 404 returned error can't find the container with id 8827fbe2975e80869c1a989eb6666bba4965efd644436f8446314dbb5436cec7 Sep 09 06:53:18.109202 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:18.109162 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" event={"ID":"628c588a-3cda-46e3-8532-8d5893ac1ef0","Type":"ContainerStarted","Data":"1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940"} Sep 09 06:53:18.109573 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:18.109216 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" event={"ID":"628c588a-3cda-46e3-8532-8d5893ac1ef0","Type":"ContainerStarted","Data":"8827fbe2975e80869c1a989eb6666bba4965efd644436f8446314dbb5436cec7"} Sep 09 06:53:18.126065 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:18.126015 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" podStartSLOduration=2.1260006 podStartE2EDuration="2.1260006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:53:18.124905159 +0000 UTC m=+1222.449715617" watchObservedRunningTime="2026-09-09 06:53:18.1260006 +0000 UTC m=+1222.450811053" Sep 09 06:53:22.418188 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.418158 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-r8424"] Sep 09 06:53:22.421134 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.421118 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:53:22.423646 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.423624 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 09 06:53:22.433065 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.433045 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-r8424"] Sep 09 06:53:22.559872 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.559844 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8717f736-fc8b-41a2-a006-3e3449eb7aed-postgresdata\") pod \"postgres-864846d4f5-r8424\" (UID: \"8717f736-fc8b-41a2-a006-3e3449eb7aed\") " pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:53:22.559997 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.559908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7v2\" (UniqueName: \"kubernetes.io/projected/8717f736-fc8b-41a2-a006-3e3449eb7aed-kube-api-access-6z7v2\") pod \"postgres-864846d4f5-r8424\" (UID: \"8717f736-fc8b-41a2-a006-3e3449eb7aed\") " pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:53:22.661257 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.661231 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7v2\" (UniqueName: \"kubernetes.io/projected/8717f736-fc8b-41a2-a006-3e3449eb7aed-kube-api-access-6z7v2\") pod \"postgres-864846d4f5-r8424\" (UID: \"8717f736-fc8b-41a2-a006-3e3449eb7aed\") " pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:53:22.661349 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.661296 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8717f736-fc8b-41a2-a006-3e3449eb7aed-postgresdata\") pod \"postgres-864846d4f5-r8424\" (UID: \"8717f736-fc8b-41a2-a006-3e3449eb7aed\") " pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:53:22.661661 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.661641 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8717f736-fc8b-41a2-a006-3e3449eb7aed-postgresdata\") pod \"postgres-864846d4f5-r8424\" (UID: \"8717f736-fc8b-41a2-a006-3e3449eb7aed\") " pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:53:22.669330 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.669270 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7v2\" (UniqueName: \"kubernetes.io/projected/8717f736-fc8b-41a2-a006-3e3449eb7aed-kube-api-access-6z7v2\") pod \"postgres-864846d4f5-r8424\" (UID: \"8717f736-fc8b-41a2-a006-3e3449eb7aed\") " pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:53:22.730768 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:22.730743 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:53:23.001145 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:23.001123 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-r8424"] Sep 09 06:53:23.003699 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:53:23.003671 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8717f736_fc8b_41a2_a006_3e3449eb7aed.slice/crio-936501f1a00b9a45d1b2706a52cb00e55e2affeb1937ba16e97ec31ffb40970b WatchSource:0}: Error finding container 936501f1a00b9a45d1b2706a52cb00e55e2affeb1937ba16e97ec31ffb40970b: Status 404 returned error can't find the container with id 936501f1a00b9a45d1b2706a52cb00e55e2affeb1937ba16e97ec31ffb40970b Sep 09 06:53:23.131013 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:23.130952 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" event={"ID":"8717f736-fc8b-41a2-a006-3e3449eb7aed","Type":"ContainerStarted","Data":"2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d"} Sep 09 06:53:23.131159 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:23.131026 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" event={"ID":"8717f736-fc8b-41a2-a006-3e3449eb7aed","Type":"ContainerStarted","Data":"936501f1a00b9a45d1b2706a52cb00e55e2affeb1937ba16e97ec31ffb40970b"} Sep 09 06:53:23.146656 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:23.146612 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" podStartSLOduration=1.146595673 podStartE2EDuration="1.146595673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:53:23.146336422 +0000 UTC m=+1227.471146879" watchObservedRunningTime="2026-09-09 06:53:23.146595673 +0000 UTC m=+1227.471406128" Sep 09 06:53:27.957700 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:27.957659 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8"] Sep 09 06:53:27.961222 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:27.961204 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:27.963646 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:27.963625 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 09 06:53:27.963764 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:27.963652 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 09 06:53:27.963764 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:27.963672 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-fcfbp\"" Sep 09 06:53:27.963764 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:27.963737 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 09 06:53:27.971660 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:27.971640 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8"] Sep 09 06:53:28.103484 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.103453 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-online-tls\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.103654 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.103494 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-registry-tls\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.103654 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.103538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72fsm\" (UniqueName: \"kubernetes.io/projected/46902d0b-31e7-4332-963b-8b9ac0ea489d-kube-api-access-72fsm\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.103654 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.103572 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46902d0b-31e7-4332-963b-8b9ac0ea489d-feast-data\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.204085 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.204052 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-registry-tls\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.204273 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.204106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72fsm\" (UniqueName: \"kubernetes.io/projected/46902d0b-31e7-4332-963b-8b9ac0ea489d-kube-api-access-72fsm\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.204273 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.204142 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46902d0b-31e7-4332-963b-8b9ac0ea489d-feast-data\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.204273 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.204171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-online-tls\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.204620 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.204599 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46902d0b-31e7-4332-963b-8b9ac0ea489d-feast-data\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.206667 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.206643 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-registry-tls\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.206769 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.206647 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-online-tls\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.211925 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.211868 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72fsm\" (UniqueName: \"kubernetes.io/projected/46902d0b-31e7-4332-963b-8b9ac0ea489d-kube-api-access-72fsm\") pod \"feast-credit-scoring-64f9d59857-2d9b8\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.271937 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.271907 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:28.549527 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:28.549505 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8"] Sep 09 06:53:28.552043 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:53:28.552011 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46902d0b_31e7_4332_963b_8b9ac0ea489d.slice/crio-71731ae13b598430acfd1f78fdef8ec9f66d81562912ad9e4d5cb587534901c1 WatchSource:0}: Error finding container 71731ae13b598430acfd1f78fdef8ec9f66d81562912ad9e4d5cb587534901c1: Status 404 returned error can't find the container with id 71731ae13b598430acfd1f78fdef8ec9f66d81562912ad9e4d5cb587534901c1 Sep 09 06:53:29.161664 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:29.161622 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerStarted","Data":"20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d"} Sep 09 06:53:29.162086 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:29.161672 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerStarted","Data":"71731ae13b598430acfd1f78fdef8ec9f66d81562912ad9e4d5cb587534901c1"} Sep 09 06:53:30.166551 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:30.166517 2563 generic.go:356] "Generic (PLEG): container finished" podID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerID="20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d" exitCode=0 Sep 09 06:53:30.166911 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:30.166568 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerDied","Data":"20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d"} Sep 09 06:53:31.172744 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:31.172709 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerStarted","Data":"92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648"} Sep 09 06:53:37.199420 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:37.199391 2563 generic.go:356] "Generic (PLEG): container finished" podID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerID="92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648" exitCode=0 Sep 09 06:53:37.199725 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:37.199459 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerDied","Data":"92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648"} Sep 09 06:53:38.206440 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:38.206402 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerStarted","Data":"245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3"} Sep 09 06:53:38.206799 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:38.206464 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerStarted","Data":"275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0"} Sep 09 06:53:38.228448 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:38.228406 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podStartSLOduration=11.228391464 podStartE2EDuration="11.228391464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:53:38.226858486 +0000 UTC m=+1242.551668943" watchObservedRunningTime="2026-09-09 06:53:38.228391464 +0000 UTC m=+1242.553201919" Sep 09 06:53:38.272328 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:38.272294 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:38.272489 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:38.272343 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:40.272535 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:40.272498 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:40.272913 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:40.272568 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:40.273867 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:40.273834 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Sep 09 06:53:40.273995 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:40.273881 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 09 06:53:43.273405 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:43.273358 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Sep 09 06:53:43.273887 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:43.273366 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 09 06:53:46.272356 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:46.272312 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Sep 09 06:53:46.280544 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:46.280517 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:46.285071 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:46.285038 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:49.273228 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:49.273190 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:49.274003 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:49.273982 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:53:53.803324 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.803294 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4"] Sep 09 06:53:53.806629 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.806612 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.808896 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.808869 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 09 06:53:53.809077 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.809057 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 09 06:53:53.809159 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.809143 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-pbd2v\"" Sep 09 06:53:53.809256 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.809241 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 09 06:53:53.818114 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.818090 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4"] Sep 09 06:53:53.818871 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.818849 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k72r8\" (UniqueName: \"kubernetes.io/projected/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-kube-api-access-k72r8\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.818985 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.818888 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-feast-data\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.818985 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.818917 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-registry-tls\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.818985 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.818935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-online-tls\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.919629 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.919604 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k72r8\" (UniqueName: \"kubernetes.io/projected/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-kube-api-access-k72r8\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.919749 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.919633 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-feast-data\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.919749 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.919661 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-registry-tls\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.919749 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.919686 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-online-tls\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.919922 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:53:53.919751 2563 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Sep 09 06:53:53.919922 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:53:53.919769 2563 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Sep 09 06:53:53.919922 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:53:53.919818 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-registry-tls podName:4f8ca83d-3be2-42c6-8bd4-26ab8f749137 nodeName:}" failed. No retries permitted until 2026-09-09 06:53:54.41980165 +0000 UTC m=+1258.744612085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-registry-tls") pod "feast-driver-ranking-86cf5858b6-9dnv4" (UID: "4f8ca83d-3be2-42c6-8bd4-26ab8f749137") : secret "feast-driver-ranking-registry-tls" not found Sep 09 06:53:53.919922 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:53:53.919832 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-online-tls podName:4f8ca83d-3be2-42c6-8bd4-26ab8f749137 nodeName:}" failed. No retries permitted until 2026-09-09 06:53:54.419826263 +0000 UTC m=+1258.744636697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-online-tls") pod "feast-driver-ranking-86cf5858b6-9dnv4" (UID: "4f8ca83d-3be2-42c6-8bd4-26ab8f749137") : secret "feast-driver-ranking-online-tls" not found Sep 09 06:53:53.920163 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.920077 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-feast-data\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:53.930727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:53.930703 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k72r8\" (UniqueName: \"kubernetes.io/projected/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-kube-api-access-k72r8\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:54.425042 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:54.425014 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-registry-tls\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:54.425042 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:54.425046 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-online-tls\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:54.427573 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:54.427551 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-registry-tls\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:54.427695 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:54.427629 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-online-tls\") pod \"feast-driver-ranking-86cf5858b6-9dnv4\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:54.715828 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:54.715741 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:53:54.989318 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:54.989297 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4"] Sep 09 06:53:54.991337 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:53:54.991307 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8ca83d_3be2_42c6_8bd4_26ab8f749137.slice/crio-0a502bd83f4013de010fdde3c8fc77318efc5b14320d98220aab80b28da2e26b WatchSource:0}: Error finding container 0a502bd83f4013de010fdde3c8fc77318efc5b14320d98220aab80b28da2e26b: Status 404 returned error can't find the container with id 0a502bd83f4013de010fdde3c8fc77318efc5b14320d98220aab80b28da2e26b Sep 09 06:53:55.285777 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:55.285685 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerStarted","Data":"6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466"} Sep 09 06:53:55.285777 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:53:55.285753 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerStarted","Data":"0a502bd83f4013de010fdde3c8fc77318efc5b14320d98220aab80b28da2e26b"} Sep 09 06:54:00.307839 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:00.307804 2563 generic.go:356] "Generic (PLEG): container finished" podID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerID="6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466" exitCode=0 Sep 09 06:54:00.308214 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:00.307874 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerDied","Data":"6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466"} Sep 09 06:54:01.313771 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:01.313732 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerStarted","Data":"9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328"} Sep 09 06:54:07.343735 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:07.343664 2563 generic.go:356] "Generic (PLEG): container finished" podID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerID="9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328" exitCode=0 Sep 09 06:54:07.343735 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:07.343717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerDied","Data":"9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328"} Sep 09 06:54:07.347406 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:07.347379 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 09 06:54:08.351610 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:08.351577 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerStarted","Data":"edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78"} Sep 09 06:54:08.351956 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:08.351625 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerStarted","Data":"00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32"} Sep 09 06:54:08.376265 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:08.376210 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podStartSLOduration=15.376190465 podStartE2EDuration="15.376190465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:54:08.373248293 +0000 UTC m=+1272.698058743" watchObservedRunningTime="2026-09-09 06:54:08.376190465 +0000 UTC m=+1272.701000922" Sep 09 06:54:09.716159 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:09.716119 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:54:09.716532 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:09.716225 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:54:09.717548 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:09.717518 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused" Sep 09 06:54:09.717644 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:09.717545 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 09 06:54:12.716407 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:12.716363 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused" Sep 09 06:54:12.716777 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:12.716370 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 09 06:54:14.716747 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:14.716711 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:54:14.717179 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:14.716811 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:54:15.716412 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:15.716373 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused" Sep 09 06:54:15.722380 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:15.722353 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:54:16.396768 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:16.396737 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:54:18.716785 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:18.716734 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused" Sep 09 06:54:21.716722 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:21.716691 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:54:21.717538 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:54:21.717520 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:55:43.795033 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:43.795004 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2"] Sep 09 06:55:43.795611 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:43.795267 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" podUID="f4d9cd43-c285-4dcc-9374-73c664f378b1" containerName="manager" containerID="cri-o://1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3" gracePeriod=10 Sep 09 06:55:44.067573 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.067550 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" Sep 09 06:55:44.128472 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.128443 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8"] Sep 09 06:55:44.128679 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.128655 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="registry" containerID="cri-o://275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0" gracePeriod=30 Sep 09 06:55:44.128751 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.128708 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="online" containerID="cri-o://245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3" gracePeriod=30 Sep 09 06:55:44.157019 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.156989 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4"] Sep 09 06:55:44.157196 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.157177 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="registry" containerID="cri-o://00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32" gracePeriod=30 Sep 09 06:55:44.157250 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.157216 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="online" containerID="cri-o://edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78" gracePeriod=30 Sep 09 06:55:44.195568 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.195536 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnx6j\" (UniqueName: \"kubernetes.io/projected/f4d9cd43-c285-4dcc-9374-73c664f378b1-kube-api-access-lnx6j\") pod \"f4d9cd43-c285-4dcc-9374-73c664f378b1\" (UID: \"f4d9cd43-c285-4dcc-9374-73c664f378b1\") " Sep 09 06:55:44.197842 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.197819 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d9cd43-c285-4dcc-9374-73c664f378b1-kube-api-access-lnx6j" (OuterVolumeSpecName: "kube-api-access-lnx6j") pod "f4d9cd43-c285-4dcc-9374-73c664f378b1" (UID: "f4d9cd43-c285-4dcc-9374-73c664f378b1"). InnerVolumeSpecName "kube-api-access-lnx6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:55:44.280247 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.280213 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-r8424"] Sep 09 06:55:44.280405 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.280385 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" podUID="8717f736-fc8b-41a2-a006-3e3449eb7aed" containerName="postgres" containerID="cri-o://2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d" gracePeriod=30 Sep 09 06:55:44.288822 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.288743 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-mh9l8"] Sep 09 06:55:44.289004 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.288904 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" podUID="628c588a-3cda-46e3-8532-8d5893ac1ef0" containerName="redis" containerID="cri-o://1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940" gracePeriod=30 Sep 09 06:55:44.297219 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.296987 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnx6j\" (UniqueName: \"kubernetes.io/projected/f4d9cd43-c285-4dcc-9374-73c664f378b1-kube-api-access-lnx6j\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:55:44.550733 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.550661 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:55:44.557181 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.557158 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" Sep 09 06:55:44.700260 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.700197 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8717f736-fc8b-41a2-a006-3e3449eb7aed-postgresdata\") pod \"8717f736-fc8b-41a2-a006-3e3449eb7aed\" (UID: \"8717f736-fc8b-41a2-a006-3e3449eb7aed\") " Sep 09 06:55:44.700493 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.700334 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z7v2\" (UniqueName: \"kubernetes.io/projected/8717f736-fc8b-41a2-a006-3e3449eb7aed-kube-api-access-6z7v2\") pod \"8717f736-fc8b-41a2-a006-3e3449eb7aed\" (UID: \"8717f736-fc8b-41a2-a006-3e3449eb7aed\") " Sep 09 06:55:44.700493 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.700364 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c97kl\" (UniqueName: \"kubernetes.io/projected/628c588a-3cda-46e3-8532-8d5893ac1ef0-kube-api-access-c97kl\") pod \"628c588a-3cda-46e3-8532-8d5893ac1ef0\" (UID: \"628c588a-3cda-46e3-8532-8d5893ac1ef0\") " Sep 09 06:55:44.702747 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.702711 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8717f736-fc8b-41a2-a006-3e3449eb7aed-kube-api-access-6z7v2" (OuterVolumeSpecName: "kube-api-access-6z7v2") pod "8717f736-fc8b-41a2-a006-3e3449eb7aed" (UID: "8717f736-fc8b-41a2-a006-3e3449eb7aed"). InnerVolumeSpecName "kube-api-access-6z7v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:55:44.702993 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.702946 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628c588a-3cda-46e3-8532-8d5893ac1ef0-kube-api-access-c97kl" (OuterVolumeSpecName: "kube-api-access-c97kl") pod "628c588a-3cda-46e3-8532-8d5893ac1ef0" (UID: "628c588a-3cda-46e3-8532-8d5893ac1ef0"). InnerVolumeSpecName "kube-api-access-c97kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:55:44.731033 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.730976 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8717f736-fc8b-41a2-a006-3e3449eb7aed-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "8717f736-fc8b-41a2-a006-3e3449eb7aed" (UID: "8717f736-fc8b-41a2-a006-3e3449eb7aed"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:55:44.801191 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.801098 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z7v2\" (UniqueName: \"kubernetes.io/projected/8717f736-fc8b-41a2-a006-3e3449eb7aed-kube-api-access-6z7v2\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:55:44.801191 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.801132 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c97kl\" (UniqueName: \"kubernetes.io/projected/628c588a-3cda-46e3-8532-8d5893ac1ef0-kube-api-access-c97kl\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:55:44.801191 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.801147 2563 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8717f736-fc8b-41a2-a006-3e3449eb7aed-postgresdata\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:55:44.804935 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.804898 2563 generic.go:356] "Generic (PLEG): container finished" podID="f4d9cd43-c285-4dcc-9374-73c664f378b1" containerID="1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3" exitCode=0 Sep 09 06:55:44.805102 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.804989 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" Sep 09 06:55:44.805164 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.804988 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" event={"ID":"f4d9cd43-c285-4dcc-9374-73c664f378b1","Type":"ContainerDied","Data":"1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3"} Sep 09 06:55:44.805214 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.805196 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2" event={"ID":"f4d9cd43-c285-4dcc-9374-73c664f378b1","Type":"ContainerDied","Data":"a548b0ee476231f0334c7da4518ca988b0eea3df3d8b4e5cc6130d3c6e5bc8ae"} Sep 09 06:55:44.805272 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.805237 2563 scope.go:117] "RemoveContainer" containerID="1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3" Sep 09 06:55:44.807466 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.807358 2563 generic.go:356] "Generic (PLEG): container finished" podID="628c588a-3cda-46e3-8532-8d5893ac1ef0" containerID="1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940" exitCode=0 Sep 09 06:55:44.807466 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.807405 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" Sep 09 06:55:44.807466 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.807440 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" event={"ID":"628c588a-3cda-46e3-8532-8d5893ac1ef0","Type":"ContainerDied","Data":"1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940"} Sep 09 06:55:44.807701 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.807647 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-mh9l8" event={"ID":"628c588a-3cda-46e3-8532-8d5893ac1ef0","Type":"ContainerDied","Data":"8827fbe2975e80869c1a989eb6666bba4965efd644436f8446314dbb5436cec7"} Sep 09 06:55:44.810395 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.809350 2563 generic.go:356] "Generic (PLEG): container finished" podID="8717f736-fc8b-41a2-a006-3e3449eb7aed" containerID="2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d" exitCode=0 Sep 09 06:55:44.810395 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.809406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" event={"ID":"8717f736-fc8b-41a2-a006-3e3449eb7aed","Type":"ContainerDied","Data":"2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d"} Sep 09 06:55:44.810395 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.809435 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" event={"ID":"8717f736-fc8b-41a2-a006-3e3449eb7aed","Type":"ContainerDied","Data":"936501f1a00b9a45d1b2706a52cb00e55e2affeb1937ba16e97ec31ffb40970b"} Sep 09 06:55:44.810395 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.809493 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-r8424" Sep 09 06:55:44.823528 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.823504 2563 scope.go:117] "RemoveContainer" containerID="1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3" Sep 09 06:55:44.823912 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:55:44.823886 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3\": container with ID starting with 1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3 not found: ID does not exist" containerID="1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3" Sep 09 06:55:44.824052 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.823924 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3"} err="failed to get container status \"1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3\": rpc error: code = NotFound desc = could not find container \"1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3\": container with ID starting with 1ce1bc4dcdded4fed4ea50265a06659e12160091465045246ed2823ea0b46bc3 not found: ID does not exist" Sep 09 06:55:44.824052 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.823945 2563 scope.go:117] "RemoveContainer" containerID="1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940" Sep 09 06:55:44.833117 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.833095 2563 scope.go:117] "RemoveContainer" containerID="1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940" Sep 09 06:55:44.833419 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:55:44.833399 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940\": container with ID starting with 1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940 not found: ID does not exist" containerID="1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940" Sep 09 06:55:44.833474 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.833430 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940"} err="failed to get container status \"1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940\": rpc error: code = NotFound desc = could not find container \"1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940\": container with ID starting with 1e04602c355993e1078e4f108594da5f83272dcb5f931cd4a37f42e1f0d8e940 not found: ID does not exist" Sep 09 06:55:44.833474 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.833446 2563 scope.go:117] "RemoveContainer" containerID="2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d" Sep 09 06:55:44.837282 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.837257 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-mh9l8"] Sep 09 06:55:44.842955 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.842903 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-mh9l8"] Sep 09 06:55:44.842955 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.843006 2563 scope.go:117] "RemoveContainer" containerID="2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d" Sep 09 06:55:44.843358 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:55:44.843330 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d\": container with ID starting with 2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d not found: ID does not exist" containerID="2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d" Sep 09 06:55:44.843497 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.843362 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d"} err="failed to get container status \"2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d\": rpc error: code = NotFound desc = could not find container \"2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d\": container with ID starting with 2a66686414815962d10f8a6b4ea5fcf887a434b918d0b8ae0093984c7e9fb55d not found: ID does not exist" Sep 09 06:55:44.853717 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.853693 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2"] Sep 09 06:55:44.857888 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.857867 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c99664db9-k2rp2"] Sep 09 06:55:44.869904 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.869878 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-r8424"] Sep 09 06:55:44.875793 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:44.875760 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-r8424"] Sep 09 06:55:45.819206 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:45.819172 2563 generic.go:356] "Generic (PLEG): container finished" podID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerID="245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3" exitCode=0 Sep 09 06:55:45.819639 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:45.819243 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerDied","Data":"245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3"} Sep 09 06:55:45.821551 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:45.821529 2563 generic.go:356] "Generic (PLEG): container finished" podID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerID="edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78" exitCode=0 Sep 09 06:55:45.821665 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:45.821566 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerDied","Data":"edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78"} Sep 09 06:55:46.147008 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:46.146947 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="628c588a-3cda-46e3-8532-8d5893ac1ef0" path="/var/lib/kubelet/pods/628c588a-3cda-46e3-8532-8d5893ac1ef0/volumes" Sep 09 06:55:46.147328 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:46.147315 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8717f736-fc8b-41a2-a006-3e3449eb7aed" path="/var/lib/kubelet/pods/8717f736-fc8b-41a2-a006-3e3449eb7aed/volumes" Sep 09 06:55:46.147586 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:46.147575 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4d9cd43-c285-4dcc-9374-73c664f378b1" path="/var/lib/kubelet/pods/f4d9cd43-c285-4dcc-9374-73c664f378b1/volumes" Sep 09 06:55:46.281474 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:46.281432 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 09 06:55:46.393903 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:46.393862 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 09 06:55:56.283233 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:56.283195 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 09 06:55:56.394071 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:55:56.394028 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 09 06:56:06.281287 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:06.281243 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 09 06:56:06.281666 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:06.281340 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:56:06.394216 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:06.394178 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Sep 09 06:56:06.394374 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:06.394277 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:56:14.774089 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.774065 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:56:14.807132 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.807114 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:56:14.826303 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.826284 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-online-tls\") pod \"46902d0b-31e7-4332-963b-8b9ac0ea489d\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " Sep 09 06:56:14.826408 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.826325 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-feast-data\") pod \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " Sep 09 06:56:14.826408 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.826357 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72fsm\" (UniqueName: \"kubernetes.io/projected/46902d0b-31e7-4332-963b-8b9ac0ea489d-kube-api-access-72fsm\") pod \"46902d0b-31e7-4332-963b-8b9ac0ea489d\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " Sep 09 06:56:14.826408 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.826383 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-registry-tls\") pod \"46902d0b-31e7-4332-963b-8b9ac0ea489d\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " Sep 09 06:56:14.826408 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.826398 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46902d0b-31e7-4332-963b-8b9ac0ea489d-feast-data\") pod \"46902d0b-31e7-4332-963b-8b9ac0ea489d\" (UID: \"46902d0b-31e7-4332-963b-8b9ac0ea489d\") " Sep 09 06:56:14.826607 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.826412 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-online-tls\") pod \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " Sep 09 06:56:14.826607 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.826431 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k72r8\" (UniqueName: \"kubernetes.io/projected/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-kube-api-access-k72r8\") pod \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " Sep 09 06:56:14.827199 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.826781 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-registry-tls\") pod \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\" (UID: \"4f8ca83d-3be2-42c6-8bd4-26ab8f749137\") " Sep 09 06:56:14.827199 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.826917 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-feast-data" (OuterVolumeSpecName: "feast-data") pod "4f8ca83d-3be2-42c6-8bd4-26ab8f749137" (UID: "4f8ca83d-3be2-42c6-8bd4-26ab8f749137"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:56:14.827199 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.827097 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:56:14.828987 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.828932 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-online-tls" (OuterVolumeSpecName: "online-tls") pod "4f8ca83d-3be2-42c6-8bd4-26ab8f749137" (UID: "4f8ca83d-3be2-42c6-8bd4-26ab8f749137"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:56:14.829347 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.829237 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46902d0b-31e7-4332-963b-8b9ac0ea489d-kube-api-access-72fsm" (OuterVolumeSpecName: "kube-api-access-72fsm") pod "46902d0b-31e7-4332-963b-8b9ac0ea489d" (UID: "46902d0b-31e7-4332-963b-8b9ac0ea489d"). InnerVolumeSpecName "kube-api-access-72fsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:56:14.829347 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.829317 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "46902d0b-31e7-4332-963b-8b9ac0ea489d" (UID: "46902d0b-31e7-4332-963b-8b9ac0ea489d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:56:14.829561 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.829531 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-online-tls" (OuterVolumeSpecName: "online-tls") pod "46902d0b-31e7-4332-963b-8b9ac0ea489d" (UID: "46902d0b-31e7-4332-963b-8b9ac0ea489d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:56:14.829561 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.829537 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-kube-api-access-k72r8" (OuterVolumeSpecName: "kube-api-access-k72r8") pod "4f8ca83d-3be2-42c6-8bd4-26ab8f749137" (UID: "4f8ca83d-3be2-42c6-8bd4-26ab8f749137"). InnerVolumeSpecName "kube-api-access-k72r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:56:14.829767 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.829745 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4f8ca83d-3be2-42c6-8bd4-26ab8f749137" (UID: "4f8ca83d-3be2-42c6-8bd4-26ab8f749137"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:56:14.837367 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.837339 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46902d0b-31e7-4332-963b-8b9ac0ea489d-feast-data" (OuterVolumeSpecName: "feast-data") pod "46902d0b-31e7-4332-963b-8b9ac0ea489d" (UID: "46902d0b-31e7-4332-963b-8b9ac0ea489d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:56:14.927458 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.927409 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:56:14.927458 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.927427 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:56:14.927458 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.927436 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72fsm\" (UniqueName: \"kubernetes.io/projected/46902d0b-31e7-4332-963b-8b9ac0ea489d-kube-api-access-72fsm\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:56:14.927458 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.927444 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46902d0b-31e7-4332-963b-8b9ac0ea489d-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:56:14.927458 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.927453 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46902d0b-31e7-4332-963b-8b9ac0ea489d-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:56:14.927458 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.927460 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:56:14.927695 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.927467 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k72r8\" (UniqueName: \"kubernetes.io/projected/4f8ca83d-3be2-42c6-8bd4-26ab8f749137-kube-api-access-k72r8\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:56:14.947649 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.947628 2563 generic.go:356] "Generic (PLEG): container finished" podID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerID="275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0" exitCode=137 Sep 09 06:56:14.947739 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.947686 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" Sep 09 06:56:14.947739 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.947696 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerDied","Data":"275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0"} Sep 09 06:56:14.947856 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.947747 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8" event={"ID":"46902d0b-31e7-4332-963b-8b9ac0ea489d","Type":"ContainerDied","Data":"71731ae13b598430acfd1f78fdef8ec9f66d81562912ad9e4d5cb587534901c1"} Sep 09 06:56:14.947856 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.947777 2563 scope.go:117] "RemoveContainer" containerID="245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3" Sep 09 06:56:14.950098 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.950075 2563 generic.go:356] "Generic (PLEG): container finished" podID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerID="00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32" exitCode=137 Sep 09 06:56:14.950187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.950139 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" Sep 09 06:56:14.950187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.950147 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerDied","Data":"00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32"} Sep 09 06:56:14.950187 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.950178 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4" event={"ID":"4f8ca83d-3be2-42c6-8bd4-26ab8f749137","Type":"ContainerDied","Data":"0a502bd83f4013de010fdde3c8fc77318efc5b14320d98220aab80b28da2e26b"} Sep 09 06:56:14.956775 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.956756 2563 scope.go:117] "RemoveContainer" containerID="275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0" Sep 09 06:56:14.965015 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.965002 2563 scope.go:117] "RemoveContainer" containerID="92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648" Sep 09 06:56:14.971511 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.971484 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8"] Sep 09 06:56:14.972412 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.972395 2563 scope.go:117] "RemoveContainer" containerID="20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d" Sep 09 06:56:14.975644 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.975625 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-64f9d59857-2d9b8"] Sep 09 06:56:14.979388 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.979373 2563 scope.go:117] "RemoveContainer" containerID="245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3" Sep 09 06:56:14.979634 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:14.979619 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3\": container with ID starting with 245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3 not found: ID does not exist" containerID="245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3" Sep 09 06:56:14.979697 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.979641 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3"} err="failed to get container status \"245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3\": rpc error: code = NotFound desc = could not find container \"245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3\": container with ID starting with 245464ab79965ebdac88eef95592e28d6e1d2d1f568dc137f777adfb7e79cbb3 not found: ID does not exist" Sep 09 06:56:14.979697 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.979656 2563 scope.go:117] "RemoveContainer" containerID="275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0" Sep 09 06:56:14.979910 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:14.979893 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0\": container with ID starting with 275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0 not found: ID does not exist" containerID="275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0" Sep 09 06:56:14.979949 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.979917 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0"} err="failed to get container status \"275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0\": rpc error: code = NotFound desc = could not find container \"275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0\": container with ID starting with 275bbfc0e908a9c57890e887b58cca1d53bec5743807266d8461e79adc6ea3b0 not found: ID does not exist" Sep 09 06:56:14.979949 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.979933 2563 scope.go:117] "RemoveContainer" containerID="92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648" Sep 09 06:56:14.980198 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:14.980181 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648\": container with ID starting with 92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648 not found: ID does not exist" containerID="92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648" Sep 09 06:56:14.980254 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.980201 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648"} err="failed to get container status \"92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648\": rpc error: code = NotFound desc = could not find container \"92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648\": container with ID starting with 92329662cac43a53648dba6fb7d68a3a14741d072bd33a51c4ec8808b48e5648 not found: ID does not exist" Sep 09 06:56:14.980254 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.980213 2563 scope.go:117] "RemoveContainer" containerID="20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d" Sep 09 06:56:14.980429 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:14.980414 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d\": container with ID starting with 20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d not found: ID does not exist" containerID="20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d" Sep 09 06:56:14.980475 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.980433 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d"} err="failed to get container status \"20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d\": rpc error: code = NotFound desc = could not find container \"20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d\": container with ID starting with 20c2db8ad98ceef14a9e06121dd9e4e0432068d515e1780966f64053d1b6c65d not found: ID does not exist" Sep 09 06:56:14.980475 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.980445 2563 scope.go:117] "RemoveContainer" containerID="edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78" Sep 09 06:56:14.986886 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.986864 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4"] Sep 09 06:56:14.987619 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.987604 2563 scope.go:117] "RemoveContainer" containerID="00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32" Sep 09 06:56:14.993157 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.993140 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-86cf5858b6-9dnv4"] Sep 09 06:56:14.995378 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:14.995365 2563 scope.go:117] "RemoveContainer" containerID="9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328" Sep 09 06:56:15.002109 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:15.002093 2563 scope.go:117] "RemoveContainer" containerID="6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466" Sep 09 06:56:15.008869 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:15.008855 2563 scope.go:117] "RemoveContainer" containerID="edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78" Sep 09 06:56:15.009219 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:15.009202 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78\": container with ID starting with edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78 not found: ID does not exist" containerID="edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78" Sep 09 06:56:15.009298 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:15.009224 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78"} err="failed to get container status \"edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78\": rpc error: code = NotFound desc = could not find container \"edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78\": container with ID starting with edaf43c7773de1f2cb1a0069c1e4eb59e4fb202edfdc6369a36d0870e39e1d78 not found: ID does not exist" Sep 09 06:56:15.009298 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:15.009239 2563 scope.go:117] "RemoveContainer" containerID="00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32" Sep 09 06:56:15.009500 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:15.009474 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32\": container with ID starting with 00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32 not found: ID does not exist" containerID="00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32" Sep 09 06:56:15.009539 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:15.009506 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32"} err="failed to get container status \"00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32\": rpc error: code = NotFound desc = could not find container \"00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32\": container with ID starting with 00e5aa7321d70ffc685110ad084199df0b97884ad88c6a50b8d327c504ea7c32 not found: ID does not exist" Sep 09 06:56:15.009539 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:15.009521 2563 scope.go:117] "RemoveContainer" containerID="9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328" Sep 09 06:56:15.009764 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:15.009748 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328\": container with ID starting with 9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328 not found: ID does not exist" containerID="9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328" Sep 09 06:56:15.009807 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:15.009767 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328"} err="failed to get container status \"9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328\": rpc error: code = NotFound desc = could not find container \"9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328\": container with ID starting with 9425f839567f1dba57ecd07465bcdeafef8f320c8fe4398c7d25b2f178411328 not found: ID does not exist" Sep 09 06:56:15.009807 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:15.009782 2563 scope.go:117] "RemoveContainer" containerID="6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466" Sep 09 06:56:15.010046 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:15.010031 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466\": container with ID starting with 6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466 not found: ID does not exist" containerID="6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466" Sep 09 06:56:15.010085 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:15.010049 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466"} err="failed to get container status \"6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466\": rpc error: code = NotFound desc = could not find container \"6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466\": container with ID starting with 6d6ccd193dc2f916c3beda8c35fb60ed115eaebd95bcef26c0f2571e1c827466 not found: ID does not exist" Sep 09 06:56:16.147499 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:16.147465 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" path="/var/lib/kubelet/pods/46902d0b-31e7-4332-963b-8b9ac0ea489d/volumes" Sep 09 06:56:16.147930 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:16.147917 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" path="/var/lib/kubelet/pods/4f8ca83d-3be2-42c6-8bd4-26ab8f749137/volumes" Sep 09 06:56:20.973281 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973248 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj"] Sep 09 06:56:20.973722 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973623 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="feast-apply" Sep 09 06:56:20.973722 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973640 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="feast-apply" Sep 09 06:56:20.973722 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973656 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="online" Sep 09 06:56:20.973722 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973665 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="online" Sep 09 06:56:20.973722 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973679 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628c588a-3cda-46e3-8532-8d5893ac1ef0" containerName="redis" Sep 09 06:56:20.973722 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973688 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="628c588a-3cda-46e3-8532-8d5893ac1ef0" containerName="redis" Sep 09 06:56:20.973722 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973698 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="feast-init" Sep 09 06:56:20.973722 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973707 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="feast-init" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973728 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="feast-init" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973735 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="feast-init" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973751 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4d9cd43-c285-4dcc-9374-73c664f378b1" containerName="manager" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973759 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4d9cd43-c285-4dcc-9374-73c664f378b1" containerName="manager" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973770 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="registry" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973778 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="registry" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973787 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="registry" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973816 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="registry" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973828 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="online" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973836 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="online" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973849 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="feast-apply" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973857 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="feast-apply" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973871 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8717f736-fc8b-41a2-a006-3e3449eb7aed" containerName="postgres" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973879 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8717f736-fc8b-41a2-a006-3e3449eb7aed" containerName="postgres" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973943 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="628c588a-3cda-46e3-8532-8d5893ac1ef0" containerName="redis" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973953 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="online" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973980 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4d9cd43-c285-4dcc-9374-73c664f378b1" containerName="manager" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973989 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="online" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.973998 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="46902d0b-31e7-4332-963b-8b9ac0ea489d" containerName="registry" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.974007 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8ca83d-3be2-42c6-8bd4-26ab8f749137" containerName="registry" Sep 09 06:56:20.974057 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.974016 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8717f736-fc8b-41a2-a006-3e3449eb7aed" containerName="postgres" Sep 09 06:56:20.979505 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.979484 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:20.985193 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.985164 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 06:56:20.985561 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.985541 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 06:56:20.985686 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.985665 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 06:56:20.985762 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.985703 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 06:56:20.985762 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.985677 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 06:56:20.985762 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.985749 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-z65tg\"" Sep 09 06:56:20.986794 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:20.986774 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 06:56:21.000473 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.000448 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj"] Sep 09 06:56:21.071006 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.070953 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.071006 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.071009 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.071285 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.071128 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.071285 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.071161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a124-2564-4211-b2b1-4809c7a718ce-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.071285 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.071196 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.071285 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.071217 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5bf4\" (UniqueName: \"kubernetes.io/projected/3f62a124-2564-4211-b2b1-4809c7a718ce-kube-api-access-n5bf4\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.171986 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.171941 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.172176 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.172007 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.172176 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.172023 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a124-2564-4211-b2b1-4809c7a718ce-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.172176 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.172044 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.172176 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.172059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5bf4\" (UniqueName: \"kubernetes.io/projected/3f62a124-2564-4211-b2b1-4809c7a718ce-kube-api-access-n5bf4\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.172176 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:21.172077 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 06:56:21.172176 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.172090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.172176 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:21.172129 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-ui-tls podName:3f62a124-2564-4211-b2b1-4809c7a718ce nodeName:}" failed. No retries permitted until 2026-09-09 06:56:21.67211457 +0000 UTC m=+1405.996925014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-rssqj" (UID: "3f62a124-2564-4211-b2b1-4809c7a718ce") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 06:56:21.172176 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:21.172172 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 06:56:21.172572 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:56:21.172228 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-offline-tls podName:3f62a124-2564-4211-b2b1-4809c7a718ce nodeName:}" failed. No retries permitted until 2026-09-09 06:56:21.672212076 +0000 UTC m=+1405.997022524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-rssqj" (UID: "3f62a124-2564-4211-b2b1-4809c7a718ce") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 06:56:21.172572 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.172522 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a124-2564-4211-b2b1-4809c7a718ce-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.174662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.174639 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.174662 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.174651 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.181577 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.181558 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5bf4\" (UniqueName: \"kubernetes.io/projected/3f62a124-2564-4211-b2b1-4809c7a718ce-kube-api-access-n5bf4\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.675727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.675687 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.675727 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.675731 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.678244 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.678222 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.678404 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.678383 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-rssqj\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:21.903515 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:21.903480 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:22.180712 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:22.180688 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj"] Sep 09 06:56:22.182554 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:56:22.182526 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f62a124_2564_4211_b2b1_4809c7a718ce.slice/crio-a3f151d2f07b9aa53cf916b27cb754e933b7f43db19b38ee76d1182918968df5 WatchSource:0}: Error finding container a3f151d2f07b9aa53cf916b27cb754e933b7f43db19b38ee76d1182918968df5: Status 404 returned error can't find the container with id a3f151d2f07b9aa53cf916b27cb754e933b7f43db19b38ee76d1182918968df5 Sep 09 06:56:22.184499 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:22.184483 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 06:56:22.990575 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:22.990453 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerStarted","Data":"a3f151d2f07b9aa53cf916b27cb754e933b7f43db19b38ee76d1182918968df5"} Sep 09 06:56:36.059608 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:36.059570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerStarted","Data":"dd5d1691c63abcf8cd0f69562e54acb9fe5cbd968a436f26566c74aea9430ee1"} Sep 09 06:56:40.077392 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:40.077316 2563 generic.go:356] "Generic (PLEG): container finished" podID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerID="dd5d1691c63abcf8cd0f69562e54acb9fe5cbd968a436f26566c74aea9430ee1" exitCode=0 Sep 09 06:56:40.077734 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:40.077386 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerDied","Data":"dd5d1691c63abcf8cd0f69562e54acb9fe5cbd968a436f26566c74aea9430ee1"} Sep 09 06:56:41.084510 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:41.084478 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerStarted","Data":"97c3d3c32a5a06758554ece3c6b7dc6424db95df07150f1b99c155b22cf1011f"} Sep 09 06:56:46.106261 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:46.106227 2563 generic.go:356] "Generic (PLEG): container finished" podID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerID="97c3d3c32a5a06758554ece3c6b7dc6424db95df07150f1b99c155b22cf1011f" exitCode=0 Sep 09 06:56:46.106709 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:46.106276 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerDied","Data":"97c3d3c32a5a06758554ece3c6b7dc6424db95df07150f1b99c155b22cf1011f"} Sep 09 06:56:47.117093 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:47.117050 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerStarted","Data":"64043bfa4aa0113b289d87e6d07eb282b7c18a814c3df65916f0e66aeafd0cf9"} Sep 09 06:56:47.117575 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:47.117112 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerStarted","Data":"d68a7dff9395ddd1eb992d5ef546f36d14ed9e2f9626009f89ee4ca4189f4b5e"} Sep 09 06:56:47.117575 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:47.117133 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerStarted","Data":"92cb57b383c906e17793c072eff0e0f9c6405e0eea3eb624a149d28d61d78d3b"} Sep 09 06:56:47.117575 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:47.117151 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerStarted","Data":"d3d336f487131ce167faacaeac95b907b4ecd4398a96b2c974a11772db89459e"} Sep 09 06:56:47.148073 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:47.148009 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podStartSLOduration=14.340751351 podStartE2EDuration="27.147987416s" totalImagesPullingTime="12.807236065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:56:47.146306783 +0000 UTC m=+1431.471117255" watchObservedRunningTime="2026-09-09 06:56:47.147987416 +0000 UTC m=+1431.472797871" Sep 09 06:56:48.904588 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:48.904545 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:48.905185 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:48.905152 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:48.905185 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:48.905186 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:48.905392 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:48.905204 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:48.906758 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:48.906722 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 09 06:56:48.906877 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:48.906743 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.39:8816: connect: connection refused" Sep 09 06:56:48.906877 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:48.906772 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.39:6571: connect: connection refused" Sep 09 06:56:48.906877 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:48.906865 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 09 06:56:51.904412 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:51.904374 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:51.905456 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:51.905004 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:51.905456 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:51.905030 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.39:8816: connect: connection refused" Sep 09 06:56:51.905456 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:51.905057 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:51.905456 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:51.905080 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:51.905456 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:51.905072 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.39:6571: connect: connection refused" Sep 09 06:56:51.905456 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:51.904705 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 09 06:56:51.905456 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:51.905326 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 09 06:56:54.905034 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:54.905003 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:54.905395 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:54.905122 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:54.905395 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:54.905348 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:54.906003 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:54.905940 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:54.906158 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:54.906098 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:54.906223 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:54.906210 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:54.910166 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:54.910150 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:56:55.165702 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:56:55.165622 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:57:57.339717 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.339689 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj"] Sep 09 06:57:57.340238 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.339934 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="registry" containerID="cri-o://d3d336f487131ce167faacaeac95b907b4ecd4398a96b2c974a11772db89459e" gracePeriod=30 Sep 09 06:57:57.340238 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.339992 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="ui" containerID="cri-o://64043bfa4aa0113b289d87e6d07eb282b7c18a814c3df65916f0e66aeafd0cf9" gracePeriod=30 Sep 09 06:57:57.340238 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.340050 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="online" containerID="cri-o://92cb57b383c906e17793c072eff0e0f9c6405e0eea3eb624a149d28d61d78d3b" gracePeriod=30 Sep 09 06:57:57.340436 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.340248 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="offline" containerID="cri-o://d68a7dff9395ddd1eb992d5ef546f36d14ed9e2f9626009f89ee4ca4189f4b5e" gracePeriod=30 Sep 09 06:57:57.760630 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.760598 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq"] Sep 09 06:57:57.763683 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.763662 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.766287 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.766253 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-2pvt5\"" Sep 09 06:57:57.780448 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.780417 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq"] Sep 09 06:57:57.835788 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.835743 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a714f2f-d0d9-4020-872e-83aa89b9ad65-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.835788 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.835797 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.836109 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.835827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.836109 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.835865 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whdtg\" (UniqueName: \"kubernetes.io/projected/8a714f2f-d0d9-4020-872e-83aa89b9ad65-kube-api-access-whdtg\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.836109 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.835890 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.836109 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.835913 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.936345 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.936293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.936345 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.936345 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whdtg\" (UniqueName: \"kubernetes.io/projected/8a714f2f-d0d9-4020-872e-83aa89b9ad65-kube-api-access-whdtg\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.936616 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.936372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.936616 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.936393 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.936616 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.936420 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a714f2f-d0d9-4020-872e-83aa89b9ad65-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.936616 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:57:57.936518 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 06:57:57.936616 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.936577 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.936616 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:57:57.936607 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-ui-tls podName:8a714f2f-d0d9-4020-872e-83aa89b9ad65 nodeName:}" failed. No retries permitted until 2026-09-09 06:57:58.436584904 +0000 UTC m=+1502.761395343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-tq7rq" (UID: "8a714f2f-d0d9-4020-872e-83aa89b9ad65") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 06:57:57.937013 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:57:57.936689 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 06:57:57.937013 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:57:57.936739 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls podName:8a714f2f-d0d9-4020-872e-83aa89b9ad65 nodeName:}" failed. No retries permitted until 2026-09-09 06:57:58.436723923 +0000 UTC m=+1502.761534356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-tq7rq" (UID: "8a714f2f-d0d9-4020-872e-83aa89b9ad65") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 06:57:57.937224 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.937197 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a714f2f-d0d9-4020-872e-83aa89b9ad65-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.939397 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.939366 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.939690 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.939674 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:57.945418 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:57.945377 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whdtg\" (UniqueName: \"kubernetes.io/projected/8a714f2f-d0d9-4020-872e-83aa89b9ad65-kube-api-access-whdtg\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:58.417522 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:58.417493 2563 generic.go:356] "Generic (PLEG): container finished" podID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerID="64043bfa4aa0113b289d87e6d07eb282b7c18a814c3df65916f0e66aeafd0cf9" exitCode=0 Sep 09 06:57:58.417522 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:58.417518 2563 generic.go:356] "Generic (PLEG): container finished" podID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerID="92cb57b383c906e17793c072eff0e0f9c6405e0eea3eb624a149d28d61d78d3b" exitCode=0 Sep 09 06:57:58.417910 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:58.417560 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerDied","Data":"64043bfa4aa0113b289d87e6d07eb282b7c18a814c3df65916f0e66aeafd0cf9"} Sep 09 06:57:58.417910 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:58.417603 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerDied","Data":"92cb57b383c906e17793c072eff0e0f9c6405e0eea3eb624a149d28d61d78d3b"} Sep 09 06:57:58.440436 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:58.440413 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:58.440554 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:58.440464 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:58.440554 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:57:58.440550 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 06:57:58.440642 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:57:58.440606 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls podName:8a714f2f-d0d9-4020-872e-83aa89b9ad65 nodeName:}" failed. No retries permitted until 2026-09-09 06:57:59.440592935 +0000 UTC m=+1503.765403369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-tq7rq" (UID: "8a714f2f-d0d9-4020-872e-83aa89b9ad65") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 06:57:58.442812 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:58.442795 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:59.446817 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:59.446785 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:59.449249 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:59.449232 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-tq7rq\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:59.572797 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:59.572762 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:57:59.849543 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:57:59.849513 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq"] Sep 09 06:57:59.850632 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:57:59.850603 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a714f2f_d0d9_4020_872e_83aa89b9ad65.slice/crio-a0ca167d9dec0429a16b8f6b9a2e3c6ee4ea1bc708f5500074e01fc29a0b9ffb WatchSource:0}: Error finding container a0ca167d9dec0429a16b8f6b9a2e3c6ee4ea1bc708f5500074e01fc29a0b9ffb: Status 404 returned error can't find the container with id a0ca167d9dec0429a16b8f6b9a2e3c6ee4ea1bc708f5500074e01fc29a0b9ffb Sep 09 06:58:00.426834 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:00.426803 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerStarted","Data":"7e65bc4eba08d84fc61035ba1d044f10a111d811c6766ebe5b8888dd141f8657"} Sep 09 06:58:00.427046 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:00.426850 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerStarted","Data":"a0ca167d9dec0429a16b8f6b9a2e3c6ee4ea1bc708f5500074e01fc29a0b9ffb"} Sep 09 06:58:04.906649 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:04.906553 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 09 06:58:05.163448 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:05.163362 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 09 06:58:05.451423 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:05.451343 2563 generic.go:356] "Generic (PLEG): container finished" podID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerID="7e65bc4eba08d84fc61035ba1d044f10a111d811c6766ebe5b8888dd141f8657" exitCode=0 Sep 09 06:58:05.451562 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:05.451419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerDied","Data":"7e65bc4eba08d84fc61035ba1d044f10a111d811c6766ebe5b8888dd141f8657"} Sep 09 06:58:06.458521 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:06.458491 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerStarted","Data":"2200daffaef3620146a249d26ba106d3f0a36379a4d6b3cdde16af4d12f16abf"} Sep 09 06:58:11.481275 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:11.481186 2563 generic.go:356] "Generic (PLEG): container finished" podID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerID="2200daffaef3620146a249d26ba106d3f0a36379a4d6b3cdde16af4d12f16abf" exitCode=0 Sep 09 06:58:11.481275 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:11.481249 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerDied","Data":"2200daffaef3620146a249d26ba106d3f0a36379a4d6b3cdde16af4d12f16abf"} Sep 09 06:58:12.492784 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:12.492746 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerStarted","Data":"b58d68bb51a047e16d52d5e38ccf5dadcd558185784d868d04909e08b46bbab8"} Sep 09 06:58:12.493310 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:12.492810 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerStarted","Data":"1c3e78d1d7b0dbdfb7c6b802a9b809d9a5d9813107183b9e2e7f689175721447"} Sep 09 06:58:12.493310 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:12.492835 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerStarted","Data":"acc3b83cb1b3fa2f7271af09bd9234e887890724d1aef08fc4df11753d5d3efe"} Sep 09 06:58:12.493310 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:12.492855 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerStarted","Data":"c436deed10091cd1e383240ecdc490c68a07201e666c1ba2c32015a83b75a45d"} Sep 09 06:58:12.525276 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:12.525206 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podStartSLOduration=15.525182939 podStartE2EDuration="15.525182939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:58:12.522974355 +0000 UTC m=+1516.847784803" watchObservedRunningTime="2026-09-09 06:58:12.525182939 +0000 UTC m=+1516.849993398" Sep 09 06:58:14.573844 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:14.573267 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:14.573844 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:14.573326 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:14.573844 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:14.573346 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:14.573844 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:14.573363 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:14.575594 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:14.575483 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 09 06:58:14.575594 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:14.575510 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.40:6571: connect: connection refused" Sep 09 06:58:14.575594 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:14.575549 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 09 06:58:14.575833 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:14.575586 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.40:8816: connect: connection refused" Sep 09 06:58:14.906661 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:14.906614 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 09 06:58:15.163099 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:15.162928 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 09 06:58:17.574146 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:17.574106 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.40:8816: connect: connection refused" Sep 09 06:58:17.574732 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:17.574106 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.40:6571: connect: connection refused" Sep 09 06:58:17.574849 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:17.574106 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 09 06:58:17.574849 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:17.574292 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 09 06:58:19.573173 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:19.573141 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:19.573524 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:19.573188 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:19.573524 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:19.573201 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:19.573524 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:19.573212 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:20.574089 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:20.574059 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:20.574435 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:20.574158 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:20.574486 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:20.574454 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:20.575304 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:20.575282 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:20.575432 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:20.575346 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:20.575432 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:20.575393 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:20.579466 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:20.579445 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:21.555819 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:21.555784 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 06:58:24.906020 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:24.905950 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Sep 09 06:58:24.907399 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:24.906145 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:58:25.162991 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:25.162900 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Sep 09 06:58:25.163123 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:25.163025 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:58:27.581904 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:27.581873 2563 generic.go:356] "Generic (PLEG): container finished" podID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerID="d68a7dff9395ddd1eb992d5ef546f36d14ed9e2f9626009f89ee4ca4189f4b5e" exitCode=137 Sep 09 06:58:27.581904 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:27.581898 2563 generic.go:356] "Generic (PLEG): container finished" podID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerID="d3d336f487131ce167faacaeac95b907b4ecd4398a96b2c974a11772db89459e" exitCode=137 Sep 09 06:58:27.582294 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:27.581954 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerDied","Data":"d68a7dff9395ddd1eb992d5ef546f36d14ed9e2f9626009f89ee4ca4189f4b5e"} Sep 09 06:58:27.582294 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:27.582028 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerDied","Data":"d3d336f487131ce167faacaeac95b907b4ecd4398a96b2c974a11772db89459e"} Sep 09 06:58:27.988712 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:27.988688 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:58:28.089389 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.089363 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-online-tls\") pod \"3f62a124-2564-4211-b2b1-4809c7a718ce\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " Sep 09 06:58:28.089567 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.089406 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a124-2564-4211-b2b1-4809c7a718ce-feast-data\") pod \"3f62a124-2564-4211-b2b1-4809c7a718ce\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " Sep 09 06:58:28.089567 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.089427 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-offline-tls\") pod \"3f62a124-2564-4211-b2b1-4809c7a718ce\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " Sep 09 06:58:28.089567 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.089451 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5bf4\" (UniqueName: \"kubernetes.io/projected/3f62a124-2564-4211-b2b1-4809c7a718ce-kube-api-access-n5bf4\") pod \"3f62a124-2564-4211-b2b1-4809c7a718ce\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " Sep 09 06:58:28.089567 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.089475 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-ui-tls\") pod \"3f62a124-2564-4211-b2b1-4809c7a718ce\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " Sep 09 06:58:28.089567 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.089492 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-registry-tls\") pod \"3f62a124-2564-4211-b2b1-4809c7a718ce\" (UID: \"3f62a124-2564-4211-b2b1-4809c7a718ce\") " Sep 09 06:58:28.089952 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.089925 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f62a124-2564-4211-b2b1-4809c7a718ce-feast-data" (OuterVolumeSpecName: "feast-data") pod "3f62a124-2564-4211-b2b1-4809c7a718ce" (UID: "3f62a124-2564-4211-b2b1-4809c7a718ce"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 06:58:28.091873 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.091838 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "3f62a124-2564-4211-b2b1-4809c7a718ce" (UID: "3f62a124-2564-4211-b2b1-4809c7a718ce"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:58:28.092133 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.092112 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f62a124-2564-4211-b2b1-4809c7a718ce-kube-api-access-n5bf4" (OuterVolumeSpecName: "kube-api-access-n5bf4") pod "3f62a124-2564-4211-b2b1-4809c7a718ce" (UID: "3f62a124-2564-4211-b2b1-4809c7a718ce"). InnerVolumeSpecName "kube-api-access-n5bf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 06:58:28.092256 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.092232 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-online-tls" (OuterVolumeSpecName: "online-tls") pod "3f62a124-2564-4211-b2b1-4809c7a718ce" (UID: "3f62a124-2564-4211-b2b1-4809c7a718ce"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:58:28.092332 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.092276 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3f62a124-2564-4211-b2b1-4809c7a718ce" (UID: "3f62a124-2564-4211-b2b1-4809c7a718ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:58:28.092332 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.092302 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "3f62a124-2564-4211-b2b1-4809c7a718ce" (UID: "3f62a124-2564-4211-b2b1-4809c7a718ce"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 06:58:28.190933 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.190905 2563 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-ui-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:58:28.190933 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.190927 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:58:28.191079 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.190956 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:58:28.191079 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.190984 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a124-2564-4211-b2b1-4809c7a718ce-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:58:28.191079 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.190992 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3f62a124-2564-4211-b2b1-4809c7a718ce-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:58:28.191079 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.191000 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5bf4\" (UniqueName: \"kubernetes.io/projected/3f62a124-2564-4211-b2b1-4809c7a718ce-kube-api-access-n5bf4\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 06:58:28.589684 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.589596 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" event={"ID":"3f62a124-2564-4211-b2b1-4809c7a718ce","Type":"ContainerDied","Data":"a3f151d2f07b9aa53cf916b27cb754e933b7f43db19b38ee76d1182918968df5"} Sep 09 06:58:28.589684 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.589664 2563 scope.go:117] "RemoveContainer" containerID="64043bfa4aa0113b289d87e6d07eb282b7c18a814c3df65916f0e66aeafd0cf9" Sep 09 06:58:28.590132 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.589610 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj" Sep 09 06:58:28.597914 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.597899 2563 scope.go:117] "RemoveContainer" containerID="d68a7dff9395ddd1eb992d5ef546f36d14ed9e2f9626009f89ee4ca4189f4b5e" Sep 09 06:58:28.605223 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.605206 2563 scope.go:117] "RemoveContainer" containerID="92cb57b383c906e17793c072eff0e0f9c6405e0eea3eb624a149d28d61d78d3b" Sep 09 06:58:28.613245 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.613128 2563 scope.go:117] "RemoveContainer" containerID="d3d336f487131ce167faacaeac95b907b4ecd4398a96b2c974a11772db89459e" Sep 09 06:58:28.616409 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.616388 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj"] Sep 09 06:58:28.620628 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.620607 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-rssqj"] Sep 09 06:58:28.621762 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.621750 2563 scope.go:117] "RemoveContainer" containerID="97c3d3c32a5a06758554ece3c6b7dc6424db95df07150f1b99c155b22cf1011f" Sep 09 06:58:28.628603 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:28.628589 2563 scope.go:117] "RemoveContainer" containerID="dd5d1691c63abcf8cd0f69562e54acb9fe5cbd968a436f26566c74aea9430ee1" Sep 09 06:58:30.147746 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:58:30.147713 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" path="/var/lib/kubelet/pods/3f62a124-2564-4211-b2b1-4809c7a718ce/volumes" Sep 09 06:59:24.958431 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958394 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq"] Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958709 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="ui" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958721 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="ui" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958737 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="offline" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958742 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="offline" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958750 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="feast-init" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958756 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="feast-init" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958763 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="online" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958768 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="online" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958775 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="registry" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958780 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="registry" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958791 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="feast-apply" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958795 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="feast-apply" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958843 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="registry" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958851 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="offline" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958857 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="ui" Sep 09 06:59:24.958900 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.958865 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f62a124-2564-4211-b2b1-4809c7a718ce" containerName="online" Sep 09 06:59:24.963623 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.963606 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:24.966365 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.966342 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 06:59:24.966829 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.966804 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 06:59:24.966829 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.966809 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 06:59:24.967058 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.966809 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 06:59:24.967058 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.966884 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 06:59:24.967820 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.967803 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-jktmh\"" Sep 09 06:59:24.967889 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.967837 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 06:59:24.974770 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:24.974745 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq"] Sep 09 06:59:25.078606 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.078586 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23c746e5-6e64-497a-b7f0-a52d3718ebfa-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.078754 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.078614 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/23c746e5-6e64-497a-b7f0-a52d3718ebfa-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.078754 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.078631 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.078754 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.078655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.078889 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.078752 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7trk\" (UniqueName: \"kubernetes.io/projected/23c746e5-6e64-497a-b7f0-a52d3718ebfa-kube-api-access-b7trk\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.078889 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.078786 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.179995 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.179950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23c746e5-6e64-497a-b7f0-a52d3718ebfa-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.180116 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.180000 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/23c746e5-6e64-497a-b7f0-a52d3718ebfa-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.180116 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.180028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.180116 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.180064 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.180116 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.180096 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7trk\" (UniqueName: \"kubernetes.io/projected/23c746e5-6e64-497a-b7f0-a52d3718ebfa-kube-api-access-b7trk\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.180313 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.180120 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.180313 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:59:25.180175 2563 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 06:59:25.180313 ip-10-0-128-45 kubenswrapper[2563]: E0909 06:59:25.180243 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-offline-tls podName:23c746e5-6e64-497a-b7f0-a52d3718ebfa nodeName:}" failed. No retries permitted until 2026-09-09 06:59:25.680220183 +0000 UTC m=+1590.005030617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-offline-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" (UID: "23c746e5-6e64-497a-b7f0-a52d3718ebfa") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 06:59:25.180445 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.180352 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23c746e5-6e64-497a-b7f0-a52d3718ebfa-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.180739 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.180718 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/23c746e5-6e64-497a-b7f0-a52d3718ebfa-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.182826 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.182799 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.182931 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.182839 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.188485 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.188463 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7trk\" (UniqueName: \"kubernetes.io/projected/23c746e5-6e64-497a-b7f0-a52d3718ebfa-kube-api-access-b7trk\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.685016 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.684970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.687561 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.687531 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-5rdhq\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:25.873492 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:25.873462 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:26.149405 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:26.149384 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq"] Sep 09 06:59:26.151054 ip-10-0-128-45 kubenswrapper[2563]: W0909 06:59:26.151025 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23c746e5_6e64_497a_b7f0_a52d3718ebfa.slice/crio-d22ceb4d321281ea73eba5dafbed0f8053b91f4052e50f98ccf9cb8c2bdf6fa8 WatchSource:0}: Error finding container d22ceb4d321281ea73eba5dafbed0f8053b91f4052e50f98ccf9cb8c2bdf6fa8: Status 404 returned error can't find the container with id d22ceb4d321281ea73eba5dafbed0f8053b91f4052e50f98ccf9cb8c2bdf6fa8 Sep 09 06:59:26.831792 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:26.831758 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerStarted","Data":"ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e"} Sep 09 06:59:26.831792 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:26.831801 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerStarted","Data":"d22ceb4d321281ea73eba5dafbed0f8053b91f4052e50f98ccf9cb8c2bdf6fa8"} Sep 09 06:59:31.854174 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:31.854143 2563 generic.go:356] "Generic (PLEG): container finished" podID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerID="ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e" exitCode=0 Sep 09 06:59:31.854559 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:31.854211 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerDied","Data":"ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e"} Sep 09 06:59:32.859931 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:32.859890 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerStarted","Data":"2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8"} Sep 09 06:59:53.955269 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:53.955236 2563 generic.go:356] "Generic (PLEG): container finished" podID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerID="2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8" exitCode=0 Sep 09 06:59:53.955641 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:53.955274 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerDied","Data":"2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8"} Sep 09 06:59:54.964176 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:54.964128 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerStarted","Data":"01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8"} Sep 09 06:59:54.964700 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:54.964203 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerStarted","Data":"a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0"} Sep 09 06:59:54.964700 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:54.964226 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerStarted","Data":"b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70"} Sep 09 06:59:54.989456 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:54.989400 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podStartSLOduration=30.989378043 podStartE2EDuration="30.989378043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:59:54.987094917 +0000 UTC m=+1619.311905374" watchObservedRunningTime="2026-09-09 06:59:54.989378043 +0000 UTC m=+1619.314188502" Sep 09 06:59:55.874510 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:55.874467 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:55.874806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:55.874533 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:55.874806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:55.874552 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:55.874806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:55.874569 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:55.874806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:55.874585 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:55.874806 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:55.874602 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 06:59:55.876087 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:55.876049 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.41:8816: connect: connection refused" Sep 09 06:59:55.876205 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:55.876088 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 09 06:59:55.876267 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:55.876205 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 09 06:59:58.874006 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:58.873933 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 09 06:59:58.874486 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:58.873955 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.41:8816: connect: connection refused" Sep 09 06:59:58.874486 ip-10-0-128-45 kubenswrapper[2563]: I0909 06:59:58.874058 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 09 07:00:01.874203 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:01.874152 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 09 07:00:01.874604 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:01.874245 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 09 07:00:01.875222 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:01.875202 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:00:01.875983 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:01.875947 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:00:04.874276 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:04.874243 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:00:04.875097 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:04.875079 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:00:05.586172 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:05.586141 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:00:06.915113 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:06.915076 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 07:00:07.919120 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:07.919071 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 07:00:17.649698 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:00:17.649661 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:01:19.239943 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:19.239855 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq"] Sep 09 07:01:19.240550 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:19.240161 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" containerID="cri-o://b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70" gracePeriod=30 Sep 09 07:01:19.240550 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:19.240386 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" containerID="cri-o://01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8" gracePeriod=30 Sep 09 07:01:19.240550 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:19.240432 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="offline" containerID="cri-o://a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0" gracePeriod=30 Sep 09 07:01:19.388497 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:19.388457 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq"] Sep 09 07:01:19.388777 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:19.388733 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="registry" containerID="cri-o://c436deed10091cd1e383240ecdc490c68a07201e666c1ba2c32015a83b75a45d" gracePeriod=30 Sep 09 07:01:19.388921 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:19.388885 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="offline" containerID="cri-o://1c3e78d1d7b0dbdfb7c6b802a9b809d9a5d9813107183b9e2e7f689175721447" gracePeriod=30 Sep 09 07:01:19.389142 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:19.388887 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="online" containerID="cri-o://acc3b83cb1b3fa2f7271af09bd9234e887890724d1aef08fc4df11753d5d3efe" gracePeriod=30 Sep 09 07:01:19.389261 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:19.388927 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="ui" containerID="cri-o://b58d68bb51a047e16d52d5e38ccf5dadcd558185784d868d04909e08b46bbab8" gracePeriod=30 Sep 09 07:01:20.341528 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:20.341496 2563 generic.go:356] "Generic (PLEG): container finished" podID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerID="01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8" exitCode=0 Sep 09 07:01:20.341528 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:20.341524 2563 generic.go:356] "Generic (PLEG): container finished" podID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerID="b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70" exitCode=0 Sep 09 07:01:20.341899 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:20.341570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerDied","Data":"01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8"} Sep 09 07:01:20.341899 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:20.341624 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerDied","Data":"b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70"} Sep 09 07:01:20.344789 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:20.344769 2563 generic.go:356] "Generic (PLEG): container finished" podID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerID="b58d68bb51a047e16d52d5e38ccf5dadcd558185784d868d04909e08b46bbab8" exitCode=0 Sep 09 07:01:20.344789 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:20.344788 2563 generic.go:356] "Generic (PLEG): container finished" podID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerID="acc3b83cb1b3fa2f7271af09bd9234e887890724d1aef08fc4df11753d5d3efe" exitCode=0 Sep 09 07:01:20.344926 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:20.344805 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerDied","Data":"b58d68bb51a047e16d52d5e38ccf5dadcd558185784d868d04909e08b46bbab8"} Sep 09 07:01:20.344926 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:20.344836 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerDied","Data":"acc3b83cb1b3fa2f7271af09bd9234e887890724d1aef08fc4df11753d5d3efe"} Sep 09 07:01:20.575710 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:20.575634 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 09 07:01:21.552872 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:21.552831 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 09 07:01:24.875098 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:24.875056 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 09 07:01:26.916493 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:26.916446 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 09 07:01:30.575267 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:30.575206 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 09 07:01:31.553259 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:31.553219 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 09 07:01:34.875690 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:34.875647 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 09 07:01:36.916017 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:36.915948 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 09 07:01:40.575802 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:40.575742 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Sep 09 07:01:40.576415 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:40.575917 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 07:01:41.552796 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:41.552756 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Sep 09 07:01:41.553000 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:41.552874 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 07:01:44.875262 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:44.875213 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Sep 09 07:01:44.875734 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:44.875376 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:01:46.915978 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:46.915932 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Sep 09 07:01:46.916364 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:46.916067 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:01:49.423651 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.423626 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:01:49.458402 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.458376 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/23c746e5-6e64-497a-b7f0-a52d3718ebfa-registry-tls\") pod \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " Sep 09 07:01:49.458544 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.458415 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-online-tls\") pod \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " Sep 09 07:01:49.458544 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.458456 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-offline-tls\") pod \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " Sep 09 07:01:49.458544 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.458479 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-ui-tls\") pod \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " Sep 09 07:01:49.458544 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.458511 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7trk\" (UniqueName: \"kubernetes.io/projected/23c746e5-6e64-497a-b7f0-a52d3718ebfa-kube-api-access-b7trk\") pod \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " Sep 09 07:01:49.458544 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.458542 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23c746e5-6e64-497a-b7f0-a52d3718ebfa-feast-data\") pod \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\" (UID: \"23c746e5-6e64-497a-b7f0-a52d3718ebfa\") " Sep 09 07:01:49.458833 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.458719 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c746e5-6e64-497a-b7f0-a52d3718ebfa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "23c746e5-6e64-497a-b7f0-a52d3718ebfa" (UID: "23c746e5-6e64-497a-b7f0-a52d3718ebfa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 07:01:49.459164 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.459133 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c746e5-6e64-497a-b7f0-a52d3718ebfa-feast-data" (OuterVolumeSpecName: "feast-data") pod "23c746e5-6e64-497a-b7f0-a52d3718ebfa" (UID: "23c746e5-6e64-497a-b7f0-a52d3718ebfa"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 07:01:49.461351 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.461310 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c746e5-6e64-497a-b7f0-a52d3718ebfa-kube-api-access-b7trk" (OuterVolumeSpecName: "kube-api-access-b7trk") pod "23c746e5-6e64-497a-b7f0-a52d3718ebfa" (UID: "23c746e5-6e64-497a-b7f0-a52d3718ebfa"). InnerVolumeSpecName "kube-api-access-b7trk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 07:01:49.461454 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.461386 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "23c746e5-6e64-497a-b7f0-a52d3718ebfa" (UID: "23c746e5-6e64-497a-b7f0-a52d3718ebfa"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:01:49.461454 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.461411 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "23c746e5-6e64-497a-b7f0-a52d3718ebfa" (UID: "23c746e5-6e64-497a-b7f0-a52d3718ebfa"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:01:49.461563 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.461504 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-online-tls" (OuterVolumeSpecName: "online-tls") pod "23c746e5-6e64-497a-b7f0-a52d3718ebfa" (UID: "23c746e5-6e64-497a-b7f0-a52d3718ebfa"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:01:49.467437 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.467416 2563 generic.go:356] "Generic (PLEG): container finished" podID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerID="a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0" exitCode=137 Sep 09 07:01:49.467526 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.467487 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" Sep 09 07:01:49.467526 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.467498 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerDied","Data":"a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0"} Sep 09 07:01:49.467641 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.467544 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq" event={"ID":"23c746e5-6e64-497a-b7f0-a52d3718ebfa","Type":"ContainerDied","Data":"d22ceb4d321281ea73eba5dafbed0f8053b91f4052e50f98ccf9cb8c2bdf6fa8"} Sep 09 07:01:49.467641 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.467571 2563 scope.go:117] "RemoveContainer" containerID="01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8" Sep 09 07:01:49.471417 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.471384 2563 generic.go:356] "Generic (PLEG): container finished" podID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerID="1c3e78d1d7b0dbdfb7c6b802a9b809d9a5d9813107183b9e2e7f689175721447" exitCode=137 Sep 09 07:01:49.471417 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.471406 2563 generic.go:356] "Generic (PLEG): container finished" podID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerID="c436deed10091cd1e383240ecdc490c68a07201e666c1ba2c32015a83b75a45d" exitCode=137 Sep 09 07:01:49.471576 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.471433 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerDied","Data":"1c3e78d1d7b0dbdfb7c6b802a9b809d9a5d9813107183b9e2e7f689175721447"} Sep 09 07:01:49.471576 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.471460 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerDied","Data":"c436deed10091cd1e383240ecdc490c68a07201e666c1ba2c32015a83b75a45d"} Sep 09 07:01:49.487156 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.487141 2563 scope.go:117] "RemoveContainer" containerID="a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0" Sep 09 07:01:49.495344 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.495328 2563 scope.go:117] "RemoveContainer" containerID="b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70" Sep 09 07:01:49.501176 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.501150 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq"] Sep 09 07:01:49.503416 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.503401 2563 scope.go:117] "RemoveContainer" containerID="2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8" Sep 09 07:01:49.510357 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.510340 2563 scope.go:117] "RemoveContainer" containerID="ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e" Sep 09 07:01:49.513174 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.513140 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-5rdhq"] Sep 09 07:01:49.517762 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.517746 2563 scope.go:117] "RemoveContainer" containerID="01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8" Sep 09 07:01:49.518054 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:01:49.518037 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8\": container with ID starting with 01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8 not found: ID does not exist" containerID="01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8" Sep 09 07:01:49.518108 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.518059 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8"} err="failed to get container status \"01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8\": rpc error: code = NotFound desc = could not find container \"01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8\": container with ID starting with 01cf80fef25eeb1f6abbe394c218fe7451dffafa4823a7f03f2fb8ab2f52a9a8 not found: ID does not exist" Sep 09 07:01:49.518108 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.518074 2563 scope.go:117] "RemoveContainer" containerID="a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0" Sep 09 07:01:49.518328 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:01:49.518311 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0\": container with ID starting with a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0 not found: ID does not exist" containerID="a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0" Sep 09 07:01:49.518373 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.518333 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0"} err="failed to get container status \"a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0\": rpc error: code = NotFound desc = could not find container \"a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0\": container with ID starting with a44740f4ed6381564f6a2631636d1355249c59b20770d0d4b0a318efcda8b6e0 not found: ID does not exist" Sep 09 07:01:49.518373 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.518347 2563 scope.go:117] "RemoveContainer" containerID="b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70" Sep 09 07:01:49.518579 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:01:49.518561 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70\": container with ID starting with b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70 not found: ID does not exist" containerID="b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70" Sep 09 07:01:49.518618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.518584 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70"} err="failed to get container status \"b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70\": rpc error: code = NotFound desc = could not find container \"b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70\": container with ID starting with b9443c41c1943ac92aa0b0c0202b7720ab271adbbd8406d185b98653091e1d70 not found: ID does not exist" Sep 09 07:01:49.518618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.518598 2563 scope.go:117] "RemoveContainer" containerID="2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8" Sep 09 07:01:49.518824 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:01:49.518806 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8\": container with ID starting with 2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8 not found: ID does not exist" containerID="2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8" Sep 09 07:01:49.518863 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.518830 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8"} err="failed to get container status \"2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8\": rpc error: code = NotFound desc = could not find container \"2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8\": container with ID starting with 2d2a74110879784b54f6fafb35483732d5dc480f022eeae0fd55e777db17eeb8 not found: ID does not exist" Sep 09 07:01:49.518863 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.518845 2563 scope.go:117] "RemoveContainer" containerID="ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e" Sep 09 07:01:49.519125 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:01:49.519110 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e\": container with ID starting with ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e not found: ID does not exist" containerID="ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e" Sep 09 07:01:49.519191 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.519127 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e"} err="failed to get container status \"ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e\": rpc error: code = NotFound desc = could not find container \"ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e\": container with ID starting with ae4852ac7ce580cdb4bd35e30d31f049e3c8e99e5af5a6af777f26e61278e67e not found: ID does not exist" Sep 09 07:01:49.560036 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.560014 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7trk\" (UniqueName: \"kubernetes.io/projected/23c746e5-6e64-497a-b7f0-a52d3718ebfa-kube-api-access-b7trk\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:49.560036 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.560032 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/23c746e5-6e64-497a-b7f0-a52d3718ebfa-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:49.560152 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.560042 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/23c746e5-6e64-497a-b7f0-a52d3718ebfa-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:49.560152 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.560068 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:49.560152 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.560076 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:49.560152 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:49.560083 2563 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/23c746e5-6e64-497a-b7f0-a52d3718ebfa-ui-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:50.028467 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.028445 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 07:01:50.064911 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.064888 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-online-tls\") pod \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " Sep 09 07:01:50.065062 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.064929 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whdtg\" (UniqueName: \"kubernetes.io/projected/8a714f2f-d0d9-4020-872e-83aa89b9ad65-kube-api-access-whdtg\") pod \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " Sep 09 07:01:50.065062 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.064971 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a714f2f-d0d9-4020-872e-83aa89b9ad65-feast-data\") pod \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " Sep 09 07:01:50.065062 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.064992 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-offline-tls\") pod \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " Sep 09 07:01:50.065062 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.065042 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls\") pod \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " Sep 09 07:01:50.065288 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.065081 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-ui-tls\") pod \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\" (UID: \"8a714f2f-d0d9-4020-872e-83aa89b9ad65\") " Sep 09 07:01:50.065511 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.065482 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a714f2f-d0d9-4020-872e-83aa89b9ad65-feast-data" (OuterVolumeSpecName: "feast-data") pod "8a714f2f-d0d9-4020-872e-83aa89b9ad65" (UID: "8a714f2f-d0d9-4020-872e-83aa89b9ad65"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 07:01:50.067164 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.067141 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a714f2f-d0d9-4020-872e-83aa89b9ad65-kube-api-access-whdtg" (OuterVolumeSpecName: "kube-api-access-whdtg") pod "8a714f2f-d0d9-4020-872e-83aa89b9ad65" (UID: "8a714f2f-d0d9-4020-872e-83aa89b9ad65"). InnerVolumeSpecName "kube-api-access-whdtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 07:01:50.067270 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.067205 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-online-tls" (OuterVolumeSpecName: "online-tls") pod "8a714f2f-d0d9-4020-872e-83aa89b9ad65" (UID: "8a714f2f-d0d9-4020-872e-83aa89b9ad65"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:01:50.067555 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.067534 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "8a714f2f-d0d9-4020-872e-83aa89b9ad65" (UID: "8a714f2f-d0d9-4020-872e-83aa89b9ad65"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:01:50.067753 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.067738 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8a714f2f-d0d9-4020-872e-83aa89b9ad65" (UID: "8a714f2f-d0d9-4020-872e-83aa89b9ad65"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:01:50.067798 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.067769 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "8a714f2f-d0d9-4020-872e-83aa89b9ad65" (UID: "8a714f2f-d0d9-4020-872e-83aa89b9ad65"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:01:50.147315 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.147294 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" path="/var/lib/kubelet/pods/23c746e5-6e64-497a-b7f0-a52d3718ebfa/volumes" Sep 09 07:01:50.166160 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.166108 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:50.166160 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.166125 2563 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-ui-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:50.166160 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.166133 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:50.166160 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.166142 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whdtg\" (UniqueName: \"kubernetes.io/projected/8a714f2f-d0d9-4020-872e-83aa89b9ad65-kube-api-access-whdtg\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:50.166160 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.166152 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a714f2f-d0d9-4020-872e-83aa89b9ad65-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:50.166160 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.166160 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8a714f2f-d0d9-4020-872e-83aa89b9ad65-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:01:50.478774 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.478699 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" Sep 09 07:01:50.479201 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.478703 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq" event={"ID":"8a714f2f-d0d9-4020-872e-83aa89b9ad65","Type":"ContainerDied","Data":"a0ca167d9dec0429a16b8f6b9a2e3c6ee4ea1bc708f5500074e01fc29a0b9ffb"} Sep 09 07:01:50.479201 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.478823 2563 scope.go:117] "RemoveContainer" containerID="b58d68bb51a047e16d52d5e38ccf5dadcd558185784d868d04909e08b46bbab8" Sep 09 07:01:50.487705 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.487684 2563 scope.go:117] "RemoveContainer" containerID="1c3e78d1d7b0dbdfb7c6b802a9b809d9a5d9813107183b9e2e7f689175721447" Sep 09 07:01:50.496176 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.496158 2563 scope.go:117] "RemoveContainer" containerID="acc3b83cb1b3fa2f7271af09bd9234e887890724d1aef08fc4df11753d5d3efe" Sep 09 07:01:50.496669 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.496649 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq"] Sep 09 07:01:50.499742 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.499718 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-tq7rq"] Sep 09 07:01:50.504165 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.504150 2563 scope.go:117] "RemoveContainer" containerID="c436deed10091cd1e383240ecdc490c68a07201e666c1ba2c32015a83b75a45d" Sep 09 07:01:50.511542 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.511526 2563 scope.go:117] "RemoveContainer" containerID="2200daffaef3620146a249d26ba106d3f0a36379a4d6b3cdde16af4d12f16abf" Sep 09 07:01:50.518497 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:50.518482 2563 scope.go:117] "RemoveContainer" containerID="7e65bc4eba08d84fc61035ba1d044f10a111d811c6766ebe5b8888dd141f8657" Sep 09 07:01:52.147083 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:01:52.147049 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" path="/var/lib/kubelet/pods/8a714f2f-d0d9-4020-872e-83aa89b9ad65/volumes" Sep 09 07:02:31.883177 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883146 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n"] Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883423 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883434 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883448 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="offline" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883454 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="offline" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883463 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="feast-init" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883469 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="feast-init" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883475 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883481 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883488 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="feast-apply" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883494 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="feast-apply" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883501 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="online" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883508 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="online" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883514 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="offline" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883519 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="offline" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883527 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="registry" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883533 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="registry" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883541 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="feast-apply" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883546 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="feast-apply" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883552 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="ui" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883557 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="ui" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883563 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="feast-init" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883568 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="feast-init" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883618 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="ui" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883625 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="offline" Sep 09 07:02:31.883618 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883635 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="online" Sep 09 07:02:31.884515 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883640 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="ui" Sep 09 07:02:31.884515 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883645 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="online" Sep 09 07:02:31.884515 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883651 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c746e5-6e64-497a-b7f0-a52d3718ebfa" containerName="offline" Sep 09 07:02:31.884515 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.883659 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a714f2f-d0d9-4020-872e-83aa89b9ad65" containerName="registry" Sep 09 07:02:31.886758 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.886742 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:31.889309 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.889288 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 07:02:31.889462 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.889323 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 07:02:31.889462 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.889364 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 07:02:31.889595 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.889288 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 07:02:31.889595 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.889468 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-tpqcf\"" Sep 09 07:02:31.889595 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.889464 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 07:02:31.889752 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.889650 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 07:02:31.896896 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.896876 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n"] Sep 09 07:02:31.947249 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.947223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfbg\" (UniqueName: \"kubernetes.io/projected/d926321f-56ba-46ee-a603-bbd52e7da4ef-kube-api-access-fnfbg\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:31.947368 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.947263 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d926321f-56ba-46ee-a603-bbd52e7da4ef-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:31.947368 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.947350 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:31.947455 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.947374 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:31.947455 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.947416 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:31.947455 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:31.947433 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.047898 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.047871 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.047898 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.047899 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.048096 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.047925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.048096 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.047941 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.048096 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.048011 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfbg\" (UniqueName: \"kubernetes.io/projected/d926321f-56ba-46ee-a603-bbd52e7da4ef-kube-api-access-fnfbg\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.048096 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:02:32.048043 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 07:02:32.048096 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:02:32.048049 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 07:02:32.048096 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.048048 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d926321f-56ba-46ee-a603-bbd52e7da4ef-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.048376 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:02:32.048113 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 07:02:32.048376 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:02:32.048114 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-online-tls podName:d926321f-56ba-46ee-a603-bbd52e7da4ef nodeName:}" failed. No retries permitted until 2026-09-09 07:02:32.548094817 +0000 UTC m=+1776.872905265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-online-tls") pod "feast-simple-feast-setup-57576b8fcf-bg57n" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 07:02:32.048376 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:02:32.048168 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-registry-tls podName:d926321f-56ba-46ee-a603-bbd52e7da4ef nodeName:}" failed. No retries permitted until 2026-09-09 07:02:32.548150738 +0000 UTC m=+1776.872961172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-bg57n" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 07:02:32.048376 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:02:32.048187 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-offline-tls podName:d926321f-56ba-46ee-a603-bbd52e7da4ef nodeName:}" failed. No retries permitted until 2026-09-09 07:02:32.548176715 +0000 UTC m=+1776.872987149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-bg57n" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 07:02:32.048376 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:02:32.048188 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 07:02:32.048376 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:02:32.048248 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-ui-tls podName:d926321f-56ba-46ee-a603-bbd52e7da4ef nodeName:}" failed. No retries permitted until 2026-09-09 07:02:32.548232624 +0000 UTC m=+1776.873043059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-bg57n" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 07:02:32.048376 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.048304 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d926321f-56ba-46ee-a603-bbd52e7da4ef-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.057633 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.057610 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfbg\" (UniqueName: \"kubernetes.io/projected/d926321f-56ba-46ee-a603-bbd52e7da4ef-kube-api-access-fnfbg\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.552794 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.552758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.552794 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.552791 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.553058 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.552819 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.553058 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.552837 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.555581 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.555556 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.555581 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.555576 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.555751 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.555656 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.555751 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.555666 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bg57n\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:32.805687 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:32.805553 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:33.083560 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:33.083534 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n"] Sep 09 07:02:33.084441 ip-10-0-128-45 kubenswrapper[2563]: W0909 07:02:33.084418 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd926321f_56ba_46ee_a603_bbd52e7da4ef.slice/crio-86039d4dfa9fcc2d1282c053a594364762bd3823800026dd12b0438763d81620 WatchSource:0}: Error finding container 86039d4dfa9fcc2d1282c053a594364762bd3823800026dd12b0438763d81620: Status 404 returned error can't find the container with id 86039d4dfa9fcc2d1282c053a594364762bd3823800026dd12b0438763d81620 Sep 09 07:02:33.654384 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:33.654352 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerStarted","Data":"8ba0e08df0e436e6b54b09659ef5d8e1e6f39622d24b1d24d7c059e6c5a0ec9a"} Sep 09 07:02:33.654559 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:33.654403 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerStarted","Data":"86039d4dfa9fcc2d1282c053a594364762bd3823800026dd12b0438763d81620"} Sep 09 07:02:38.674688 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:38.674653 2563 generic.go:356] "Generic (PLEG): container finished" podID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerID="8ba0e08df0e436e6b54b09659ef5d8e1e6f39622d24b1d24d7c059e6c5a0ec9a" exitCode=0 Sep 09 07:02:38.675174 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:38.674727 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerDied","Data":"8ba0e08df0e436e6b54b09659ef5d8e1e6f39622d24b1d24d7c059e6c5a0ec9a"} Sep 09 07:02:39.681698 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:39.681662 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerStarted","Data":"0c740b33e802c714ead8b91b94b0bf044f9370e2402ac99b11b5247c765e025e"} Sep 09 07:02:44.704988 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:44.704893 2563 generic.go:356] "Generic (PLEG): container finished" podID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerID="0c740b33e802c714ead8b91b94b0bf044f9370e2402ac99b11b5247c765e025e" exitCode=0 Sep 09 07:02:44.704988 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:44.704974 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerDied","Data":"0c740b33e802c714ead8b91b94b0bf044f9370e2402ac99b11b5247c765e025e"} Sep 09 07:02:45.715373 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:45.715334 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerStarted","Data":"8d60bb15ffb14e334c335d0b1589a4eff819f4591bc422608caa8de1d0729433"} Sep 09 07:02:45.715866 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:45.715396 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerStarted","Data":"c6647611332cb5b4b227af0f857a28c8a0278975178ddc75c9ce9008931f3675"} Sep 09 07:02:45.715866 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:45.715419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerStarted","Data":"fab7a5c89957ad60b4746167e84e186f5e33e40f625ca61bfee89543064566ff"} Sep 09 07:02:45.715866 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:45.715440 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerStarted","Data":"0c5e8fe0b730d67bfa2b1228846b68f408b642169561a1ad4a558df4d7e2e692"} Sep 09 07:02:45.741328 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:45.741258 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podStartSLOduration=14.741237006 podStartE2EDuration="14.741237006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:02:45.737728883 +0000 UTC m=+1790.062539340" watchObservedRunningTime="2026-09-09 07:02:45.741237006 +0000 UTC m=+1790.066047463" Sep 09 07:02:47.806580 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:47.806536 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:47.807076 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:47.806606 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:47.807076 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:47.806627 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:47.807367 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:47.807339 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:47.808532 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:47.808497 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.42:6571: connect: connection refused" Sep 09 07:02:47.808532 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:47.808517 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.42:8816: connect: connection refused" Sep 09 07:02:47.808725 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:47.808517 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 09 07:02:47.808725 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:47.808569 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 09 07:02:50.805936 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:50.805888 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.42:8816: connect: connection refused" Sep 09 07:02:50.806468 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:50.805891 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.42:6571: connect: connection refused" Sep 09 07:02:50.806468 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:50.805888 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 09 07:02:50.806468 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:50.806204 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 09 07:02:52.805728 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:52.805688 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:52.806230 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:52.805746 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:52.806230 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:52.805767 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:52.806230 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:52.805783 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:53.807205 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:53.807172 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:53.807667 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:53.807417 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:53.807734 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:53.807712 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:53.812943 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:53.812920 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:54.765085 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:54.764986 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:54.765258 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:54.765218 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:54.767105 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:54.767085 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:02:54.768074 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:02:54.768057 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:03:56.651246 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:56.651209 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n"] Sep 09 07:03:56.651850 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:56.651441 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="registry" containerID="cri-o://0c5e8fe0b730d67bfa2b1228846b68f408b642169561a1ad4a558df4d7e2e692" gracePeriod=30 Sep 09 07:03:56.651850 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:56.651518 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="offline" containerID="cri-o://c6647611332cb5b4b227af0f857a28c8a0278975178ddc75c9ce9008931f3675" gracePeriod=30 Sep 09 07:03:56.651850 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:56.651518 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="ui" containerID="cri-o://8d60bb15ffb14e334c335d0b1589a4eff819f4591bc422608caa8de1d0729433" gracePeriod=30 Sep 09 07:03:56.651850 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:56.651509 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="online" containerID="cri-o://fab7a5c89957ad60b4746167e84e186f5e33e40f625ca61bfee89543064566ff" gracePeriod=30 Sep 09 07:03:57.040350 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.040249 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr"] Sep 09 07:03:57.044208 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.044182 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.046793 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.046768 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-xzgvw\"" Sep 09 07:03:57.055271 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.055240 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr"] Sep 09 07:03:57.092242 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.092197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6sr\" (UniqueName: \"kubernetes.io/projected/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-kube-api-access-jq6sr\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.092242 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.092238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.092534 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.092332 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.092534 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.092389 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.092534 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.092419 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.092534 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.092475 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.193658 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.193628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.193802 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.193667 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.194084 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.193894 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.194232 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.194129 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.194628 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.194598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6sr\" (UniqueName: \"kubernetes.io/projected/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-kube-api-access-jq6sr\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.194954 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.194933 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.195228 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.195207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.195484 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:03:57.195463 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 07:03:57.195645 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:03:57.195631 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-ui-tls podName:6e842f35-4b92-4a2c-b34d-7bf05c0353d4 nodeName:}" failed. No retries permitted until 2026-09-09 07:03:57.695610786 +0000 UTC m=+1862.020421223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-fjhkr" (UID: "6e842f35-4b92-4a2c-b34d-7bf05c0353d4") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 07:03:57.196145 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:03:57.196126 2563 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 07:03:57.196359 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:03:57.196339 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-offline-tls podName:6e842f35-4b92-4a2c-b34d-7bf05c0353d4 nodeName:}" failed. No retries permitted until 2026-09-09 07:03:57.696318233 +0000 UTC m=+1862.021128682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-fjhkr" (UID: "6e842f35-4b92-4a2c-b34d-7bf05c0353d4") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 07:03:57.196903 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.196871 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.197015 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.196909 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.203630 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.203606 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6sr\" (UniqueName: \"kubernetes.io/projected/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-kube-api-access-jq6sr\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.699429 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.699352 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.699429 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.699401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.701909 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.701888 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.702005 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.701916 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fjhkr\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:57.956885 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:57.956814 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:03:58.025799 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:58.025759 2563 generic.go:356] "Generic (PLEG): container finished" podID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerID="8d60bb15ffb14e334c335d0b1589a4eff819f4591bc422608caa8de1d0729433" exitCode=0 Sep 09 07:03:58.025799 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:58.025786 2563 generic.go:356] "Generic (PLEG): container finished" podID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerID="fab7a5c89957ad60b4746167e84e186f5e33e40f625ca61bfee89543064566ff" exitCode=0 Sep 09 07:03:58.025990 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:58.025798 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerDied","Data":"8d60bb15ffb14e334c335d0b1589a4eff819f4591bc422608caa8de1d0729433"} Sep 09 07:03:58.025990 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:58.025853 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerDied","Data":"fab7a5c89957ad60b4746167e84e186f5e33e40f625ca61bfee89543064566ff"} Sep 09 07:03:58.233698 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:58.233674 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr"] Sep 09 07:03:58.235417 ip-10-0-128-45 kubenswrapper[2563]: W0909 07:03:58.235387 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e842f35_4b92_4a2c_b34d_7bf05c0353d4.slice/crio-af914a0409eda0d37e37de47bd2a2a73c7e6db0780f4c08323351c541f907039 WatchSource:0}: Error finding container af914a0409eda0d37e37de47bd2a2a73c7e6db0780f4c08323351c541f907039: Status 404 returned error can't find the container with id af914a0409eda0d37e37de47bd2a2a73c7e6db0780f4c08323351c541f907039 Sep 09 07:03:59.033010 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:59.032954 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerStarted","Data":"c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee"} Sep 09 07:03:59.033482 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:03:59.033019 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerStarted","Data":"af914a0409eda0d37e37de47bd2a2a73c7e6db0780f4c08323351c541f907039"} Sep 09 07:04:04.054143 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:04.054105 2563 generic.go:356] "Generic (PLEG): container finished" podID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerID="c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee" exitCode=0 Sep 09 07:04:04.054538 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:04.054180 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerDied","Data":"c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee"} Sep 09 07:04:04.764715 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:04.764666 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 09 07:04:04.764715 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:04.764691 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 09 07:04:05.060275 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:05.060196 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerStarted","Data":"031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b"} Sep 09 07:04:10.082300 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:10.082211 2563 generic.go:356] "Generic (PLEG): container finished" podID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerID="031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b" exitCode=0 Sep 09 07:04:10.082300 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:10.082253 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerDied","Data":"031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b"} Sep 09 07:04:11.092778 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:11.092742 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerStarted","Data":"c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16"} Sep 09 07:04:11.093344 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:11.093311 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerStarted","Data":"ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47"} Sep 09 07:04:11.093344 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:11.093352 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerStarted","Data":"775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673"} Sep 09 07:04:11.093545 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:11.093368 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerStarted","Data":"3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669"} Sep 09 07:04:11.124336 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:11.124263 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podStartSLOduration=14.124240934 podStartE2EDuration="14.124240934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:04:11.122075385 +0000 UTC m=+1875.446885867" watchObservedRunningTime="2026-09-09 07:04:11.124240934 +0000 UTC m=+1875.449051392" Sep 09 07:04:12.957330 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:12.957287 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:12.957802 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:12.957354 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:12.957802 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:12.957482 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:12.957802 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:12.957588 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:12.959794 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:12.959752 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="online" probeResult="failure" output="Get \"https://10.132.0.43:6567/health\": dial tcp 10.132.0.43:6567: connect: connection refused" Sep 09 07:04:12.959918 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:12.959790 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.43:8443: connect: connection refused" Sep 09 07:04:12.959918 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:12.959876 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.43:6571: connect: connection refused" Sep 09 07:04:12.960716 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:12.960683 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.43:8816: connect: connection refused" Sep 09 07:04:14.765128 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:14.764900 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 09 07:04:14.765128 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:14.764910 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 09 07:04:15.957413 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:15.957371 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.43:8816: connect: connection refused" Sep 09 07:04:15.958004 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:15.957387 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.43:8443: connect: connection refused" Sep 09 07:04:15.958197 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:15.957439 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.43:6571: connect: connection refused" Sep 09 07:04:15.958330 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:15.957513 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="online" probeResult="failure" output="Get \"https://10.132.0.43:6567/health\": dial tcp 10.132.0.43:6567: connect: connection refused" Sep 09 07:04:17.957870 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:17.957785 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:17.957870 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:17.957858 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:17.957870 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:17.957874 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:17.958327 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:17.957885 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:18.958106 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:18.958026 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:18.958585 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:18.958246 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:18.958585 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:18.958351 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:18.963325 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:18.963304 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:19.140149 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:19.140113 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:19.140369 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:19.140201 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:19.140369 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:19.140351 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:19.143145 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:19.143123 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:04:24.765499 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:24.765442 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="online" probeResult="failure" output="Get \"https://10.132.0.42:6567/health\": dial tcp 10.132.0.42:6567: connect: connection refused" Sep 09 07:04:24.765982 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:24.765549 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.42:8443: connect: connection refused" Sep 09 07:04:24.765982 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:24.765612 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:04:24.765982 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:24.765681 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:04:27.179689 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.179659 2563 generic.go:356] "Generic (PLEG): container finished" podID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerID="c6647611332cb5b4b227af0f857a28c8a0278975178ddc75c9ce9008931f3675" exitCode=137 Sep 09 07:04:27.179689 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.179684 2563 generic.go:356] "Generic (PLEG): container finished" podID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerID="0c5e8fe0b730d67bfa2b1228846b68f408b642169561a1ad4a558df4d7e2e692" exitCode=137 Sep 09 07:04:27.180146 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.179733 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerDied","Data":"c6647611332cb5b4b227af0f857a28c8a0278975178ddc75c9ce9008931f3675"} Sep 09 07:04:27.180146 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.179774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerDied","Data":"0c5e8fe0b730d67bfa2b1228846b68f408b642169561a1ad4a558df4d7e2e692"} Sep 09 07:04:27.315469 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.315443 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:04:27.342079 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.342049 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-registry-tls\") pod \"d926321f-56ba-46ee-a603-bbd52e7da4ef\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " Sep 09 07:04:27.342262 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.342093 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d926321f-56ba-46ee-a603-bbd52e7da4ef-feast-data\") pod \"d926321f-56ba-46ee-a603-bbd52e7da4ef\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " Sep 09 07:04:27.342262 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.342130 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-offline-tls\") pod \"d926321f-56ba-46ee-a603-bbd52e7da4ef\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " Sep 09 07:04:27.342262 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.342182 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-ui-tls\") pod \"d926321f-56ba-46ee-a603-bbd52e7da4ef\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " Sep 09 07:04:27.342262 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.342225 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnfbg\" (UniqueName: \"kubernetes.io/projected/d926321f-56ba-46ee-a603-bbd52e7da4ef-kube-api-access-fnfbg\") pod \"d926321f-56ba-46ee-a603-bbd52e7da4ef\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " Sep 09 07:04:27.342262 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.342246 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-online-tls\") pod \"d926321f-56ba-46ee-a603-bbd52e7da4ef\" (UID: \"d926321f-56ba-46ee-a603-bbd52e7da4ef\") " Sep 09 07:04:27.343233 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.343023 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926321f-56ba-46ee-a603-bbd52e7da4ef-feast-data" (OuterVolumeSpecName: "feast-data") pod "d926321f-56ba-46ee-a603-bbd52e7da4ef" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 07:04:27.345701 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.345673 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d926321f-56ba-46ee-a603-bbd52e7da4ef" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:04:27.345825 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.345684 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d926321f-56ba-46ee-a603-bbd52e7da4ef" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:04:27.345825 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.345812 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d926321f-56ba-46ee-a603-bbd52e7da4ef" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:04:27.345938 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.345896 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d926321f-56ba-46ee-a603-bbd52e7da4ef-kube-api-access-fnfbg" (OuterVolumeSpecName: "kube-api-access-fnfbg") pod "d926321f-56ba-46ee-a603-bbd52e7da4ef" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef"). InnerVolumeSpecName "kube-api-access-fnfbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 07:04:27.346244 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.346226 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-online-tls" (OuterVolumeSpecName: "online-tls") pod "d926321f-56ba-46ee-a603-bbd52e7da4ef" (UID: "d926321f-56ba-46ee-a603-bbd52e7da4ef"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:04:27.443864 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.443787 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:04:27.443864 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.443812 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d926321f-56ba-46ee-a603-bbd52e7da4ef-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:04:27.443864 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.443821 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:04:27.443864 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.443829 2563 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-ui-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:04:27.443864 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.443837 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnfbg\" (UniqueName: \"kubernetes.io/projected/d926321f-56ba-46ee-a603-bbd52e7da4ef-kube-api-access-fnfbg\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:04:27.443864 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:27.443866 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d926321f-56ba-46ee-a603-bbd52e7da4ef-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:04:28.187226 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.187197 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" Sep 09 07:04:28.187693 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.187198 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n" event={"ID":"d926321f-56ba-46ee-a603-bbd52e7da4ef","Type":"ContainerDied","Data":"86039d4dfa9fcc2d1282c053a594364762bd3823800026dd12b0438763d81620"} Sep 09 07:04:28.187693 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.187336 2563 scope.go:117] "RemoveContainer" containerID="8d60bb15ffb14e334c335d0b1589a4eff819f4591bc422608caa8de1d0729433" Sep 09 07:04:28.195851 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.195832 2563 scope.go:117] "RemoveContainer" containerID="c6647611332cb5b4b227af0f857a28c8a0278975178ddc75c9ce9008931f3675" Sep 09 07:04:28.203797 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.203779 2563 scope.go:117] "RemoveContainer" containerID="fab7a5c89957ad60b4746167e84e186f5e33e40f625ca61bfee89543064566ff" Sep 09 07:04:28.210411 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.210389 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n"] Sep 09 07:04:28.211903 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.211887 2563 scope.go:117] "RemoveContainer" containerID="0c5e8fe0b730d67bfa2b1228846b68f408b642169561a1ad4a558df4d7e2e692" Sep 09 07:04:28.217070 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.217046 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bg57n"] Sep 09 07:04:28.220563 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.220546 2563 scope.go:117] "RemoveContainer" containerID="0c740b33e802c714ead8b91b94b0bf044f9370e2402ac99b11b5247c765e025e" Sep 09 07:04:28.227817 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:28.227800 2563 scope.go:117] "RemoveContainer" containerID="8ba0e08df0e436e6b54b09659ef5d8e1e6f39622d24b1d24d7c059e6c5a0ec9a" Sep 09 07:04:30.147349 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:04:30.147314 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" path="/var/lib/kubelet/pods/d926321f-56ba-46ee-a603-bbd52e7da4ef/volumes" Sep 09 07:05:21.729478 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.729442 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47"] Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730782 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="feast-apply" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730809 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="feast-apply" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730839 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="ui" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730847 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="ui" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730862 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="online" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730869 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="online" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730897 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="feast-init" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730905 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="feast-init" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730913 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="offline" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730921 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="offline" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730944 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="registry" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.730952 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="registry" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.731133 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="ui" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.731145 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="online" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.731169 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="registry" Sep 09 07:05:21.732930 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.731179 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d926321f-56ba-46ee-a603-bbd52e7da4ef" containerName="offline" Sep 09 07:05:21.735895 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.735873 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.738577 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.738553 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 07:05:21.738677 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.738608 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 07:05:21.738677 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.738650 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 07:05:21.738786 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.738759 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-cp8m4\"" Sep 09 07:05:21.738914 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.738896 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 07:05:21.739006 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.738901 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 07:05:21.739006 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.738902 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 07:05:21.744861 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.744844 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47"] Sep 09 07:05:21.875070 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.875034 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.875244 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.875082 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3172c7c9-d194-4f02-ac09-9dd878f7df79-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.875244 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.875138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3172c7c9-d194-4f02-ac09-9dd878f7df79-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.875244 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.875218 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2gwx\" (UniqueName: \"kubernetes.io/projected/3172c7c9-d194-4f02-ac09-9dd878f7df79-kube-api-access-l2gwx\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.875244 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.875240 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.875385 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.875299 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.976512 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.976478 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2gwx\" (UniqueName: \"kubernetes.io/projected/3172c7c9-d194-4f02-ac09-9dd878f7df79-kube-api-access-l2gwx\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.976512 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.976513 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.976769 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.976537 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.976769 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.976560 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.976769 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.976682 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3172c7c9-d194-4f02-ac09-9dd878f7df79-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.976769 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.976718 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3172c7c9-d194-4f02-ac09-9dd878f7df79-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.977175 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.977151 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3172c7c9-d194-4f02-ac09-9dd878f7df79-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.977581 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.977454 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3172c7c9-d194-4f02-ac09-9dd878f7df79-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.979341 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.979313 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.979428 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.979400 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.979503 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.979442 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:21.985472 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:21.985454 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2gwx\" (UniqueName: \"kubernetes.io/projected/3172c7c9-d194-4f02-ac09-9dd878f7df79-kube-api-access-l2gwx\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-28v47\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:22.046153 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:22.046124 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:22.323367 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:22.323286 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47"] Sep 09 07:05:22.327012 ip-10-0-128-45 kubenswrapper[2563]: W0909 07:05:22.326985 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3172c7c9_d194_4f02_ac09_9dd878f7df79.slice/crio-f307afc063f76c6bea1d7b7cba1a4dc752f3f7cd59f404b5db60ed92c0ef0b70 WatchSource:0}: Error finding container f307afc063f76c6bea1d7b7cba1a4dc752f3f7cd59f404b5db60ed92c0ef0b70: Status 404 returned error can't find the container with id f307afc063f76c6bea1d7b7cba1a4dc752f3f7cd59f404b5db60ed92c0ef0b70 Sep 09 07:05:22.402202 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:22.402169 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerStarted","Data":"2fa60b25f815832ab58467a1845f83a7cca35047366320a56b0e783b81cef454"} Sep 09 07:05:22.402300 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:22.402214 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerStarted","Data":"f307afc063f76c6bea1d7b7cba1a4dc752f3f7cd59f404b5db60ed92c0ef0b70"} Sep 09 07:05:27.424869 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:27.424831 2563 generic.go:356] "Generic (PLEG): container finished" podID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerID="2fa60b25f815832ab58467a1845f83a7cca35047366320a56b0e783b81cef454" exitCode=0 Sep 09 07:05:27.425269 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:27.424905 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerDied","Data":"2fa60b25f815832ab58467a1845f83a7cca35047366320a56b0e783b81cef454"} Sep 09 07:05:28.431392 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:28.431361 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerStarted","Data":"d5ef4ea3b8e3c9266ad2f398040567b5c236103775719fadc4aa07329e2ec218"} Sep 09 07:05:54.540712 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:54.540633 2563 generic.go:356] "Generic (PLEG): container finished" podID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerID="d5ef4ea3b8e3c9266ad2f398040567b5c236103775719fadc4aa07329e2ec218" exitCode=0 Sep 09 07:05:54.540712 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:54.540696 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerDied","Data":"d5ef4ea3b8e3c9266ad2f398040567b5c236103775719fadc4aa07329e2ec218"} Sep 09 07:05:55.550123 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:55.550080 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerStarted","Data":"a4dd96f411b8035dcca086efe416ab3a635ede7639501fb14b04d22ad818b4a1"} Sep 09 07:05:55.550621 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:55.550144 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerStarted","Data":"8221d1d33586be6f9b7057cf097a695556a8e3203bd1c93cc4211d26de7fed1f"} Sep 09 07:05:55.550621 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:55.550167 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerStarted","Data":"f6463919b4a79930b810301959df87b4453e0f6fd8a73597216c1ca5b312e2a5"} Sep 09 07:05:55.583420 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:55.583356 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podStartSLOduration=34.58334155 podStartE2EDuration="34.58334155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:05:55.583289678 +0000 UTC m=+1979.908100169" watchObservedRunningTime="2026-09-09 07:05:55.58334155 +0000 UTC m=+1979.908152006" Sep 09 07:05:58.047050 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:58.047009 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:58.047540 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:58.047079 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:58.047540 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:58.047099 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:05:58.048694 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:58.048649 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Sep 09 07:05:58.048809 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:58.048681 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.44:8816: connect: connection refused" Sep 09 07:05:58.048809 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:05:58.048684 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": dial tcp 10.132.0.44:6567: connect: connection refused" Sep 09 07:06:01.046825 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:01.046778 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Sep 09 07:06:01.047322 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:01.046788 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.44:8816: connect: connection refused" Sep 09 07:06:01.047322 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:01.046873 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": dial tcp 10.132.0.44:6567: connect: connection refused" Sep 09 07:06:02.046735 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:02.046704 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:06:02.046934 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:02.046792 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:06:02.046934 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:02.046839 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:06:04.047843 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:04.047807 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:06:04.048313 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:04.048003 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:06:04.048761 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:04.048741 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:06:04.048920 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:04.048866 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:06:05.047223 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:05.047169 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 09 07:06:08.051105 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:08.051067 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 07:06:11.051129 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:11.051087 2563 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 07:06:13.878020 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:13.877988 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:06:14.758327 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:06:14.758290 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:07:15.868782 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:15.868750 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47"] Sep 09 07:07:15.869362 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:15.868987 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" containerID="cri-o://f6463919b4a79930b810301959df87b4453e0f6fd8a73597216c1ca5b312e2a5" gracePeriod=30 Sep 09 07:07:15.869362 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:15.869063 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="ui" containerID="cri-o://a4dd96f411b8035dcca086efe416ab3a635ede7639501fb14b04d22ad818b4a1" gracePeriod=30 Sep 09 07:07:15.869362 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:15.869080 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="offline" containerID="cri-o://8221d1d33586be6f9b7057cf097a695556a8e3203bd1c93cc4211d26de7fed1f" gracePeriod=30 Sep 09 07:07:16.000834 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.000803 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr"] Sep 09 07:07:16.001324 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.001117 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="registry" containerID="cri-o://3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669" gracePeriod=30 Sep 09 07:07:16.001324 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.001198 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="online" containerID="cri-o://775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673" gracePeriod=30 Sep 09 07:07:16.001324 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.001187 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="offline" containerID="cri-o://ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47" gracePeriod=30 Sep 09 07:07:16.001324 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.001188 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="ui" containerID="cri-o://c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16" gracePeriod=30 Sep 09 07:07:16.892942 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.892911 2563 generic.go:356] "Generic (PLEG): container finished" podID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerID="c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16" exitCode=0 Sep 09 07:07:16.892942 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.892938 2563 generic.go:356] "Generic (PLEG): container finished" podID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerID="775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673" exitCode=0 Sep 09 07:07:16.893379 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.892992 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerDied","Data":"c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16"} Sep 09 07:07:16.893379 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.893038 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerDied","Data":"775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673"} Sep 09 07:07:16.895707 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.895681 2563 generic.go:356] "Generic (PLEG): container finished" podID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerID="a4dd96f411b8035dcca086efe416ab3a635ede7639501fb14b04d22ad818b4a1" exitCode=0 Sep 09 07:07:16.895707 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.895700 2563 generic.go:356] "Generic (PLEG): container finished" podID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerID="f6463919b4a79930b810301959df87b4453e0f6fd8a73597216c1ca5b312e2a5" exitCode=0 Sep 09 07:07:16.895879 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.895757 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerDied","Data":"a4dd96f411b8035dcca086efe416ab3a635ede7639501fb14b04d22ad818b4a1"} Sep 09 07:07:16.895879 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:16.895802 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerDied","Data":"f6463919b4a79930b810301959df87b4453e0f6fd8a73597216c1ca5b312e2a5"} Sep 09 07:07:19.140052 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:19.139923 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.43:8443: connect: connection refused" Sep 09 07:07:19.140052 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:19.140000 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="online" probeResult="failure" output="Get \"https://10.132.0.43:6567/health\": dial tcp 10.132.0.43:6567: connect: connection refused" Sep 09 07:07:23.878923 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:23.878877 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": dial tcp 10.132.0.44:6567: connect: connection refused" Sep 09 07:07:24.048947 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:24.048909 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Sep 09 07:07:29.140754 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:29.140703 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.43:8443: connect: connection refused" Sep 09 07:07:29.140754 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:29.140740 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="online" probeResult="failure" output="Get \"https://10.132.0.43:6567/health\": dial tcp 10.132.0.43:6567: connect: connection refused" Sep 09 07:07:33.878657 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:33.878612 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": dial tcp 10.132.0.44:6567: connect: connection refused" Sep 09 07:07:34.049259 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:34.049217 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Sep 09 07:07:39.140878 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:39.140826 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.43:8443: connect: connection refused" Sep 09 07:07:39.140878 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:39.140858 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="online" probeResult="failure" output="Get \"https://10.132.0.43:6567/health\": dial tcp 10.132.0.43:6567: connect: connection refused" Sep 09 07:07:39.141524 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:39.140998 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:07:39.141524 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:39.141068 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:07:43.878761 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:43.878719 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": dial tcp 10.132.0.44:6567: connect: connection refused" Sep 09 07:07:43.879175 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:43.878846 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:07:44.048716 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:44.048681 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Sep 09 07:07:44.048905 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:44.048796 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:07:46.019265 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.019238 2563 generic.go:356] "Generic (PLEG): container finished" podID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerID="8221d1d33586be6f9b7057cf097a695556a8e3203bd1c93cc4211d26de7fed1f" exitCode=137 Sep 09 07:07:46.019658 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.019315 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerDied","Data":"8221d1d33586be6f9b7057cf097a695556a8e3203bd1c93cc4211d26de7fed1f"} Sep 09 07:07:46.514102 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.514079 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:07:46.605437 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.605400 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3172c7c9-d194-4f02-ac09-9dd878f7df79-registry-tls\") pod \"3172c7c9-d194-4f02-ac09-9dd878f7df79\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " Sep 09 07:07:46.605573 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.605467 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-offline-tls\") pod \"3172c7c9-d194-4f02-ac09-9dd878f7df79\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " Sep 09 07:07:46.605573 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.605514 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3172c7c9-d194-4f02-ac09-9dd878f7df79-feast-data\") pod \"3172c7c9-d194-4f02-ac09-9dd878f7df79\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " Sep 09 07:07:46.605573 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.605547 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-online-tls\") pod \"3172c7c9-d194-4f02-ac09-9dd878f7df79\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " Sep 09 07:07:46.605722 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.605589 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-ui-tls\") pod \"3172c7c9-d194-4f02-ac09-9dd878f7df79\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " Sep 09 07:07:46.605722 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.605623 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2gwx\" (UniqueName: \"kubernetes.io/projected/3172c7c9-d194-4f02-ac09-9dd878f7df79-kube-api-access-l2gwx\") pod \"3172c7c9-d194-4f02-ac09-9dd878f7df79\" (UID: \"3172c7c9-d194-4f02-ac09-9dd878f7df79\") " Sep 09 07:07:46.605833 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.605815 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3172c7c9-d194-4f02-ac09-9dd878f7df79-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3172c7c9-d194-4f02-ac09-9dd878f7df79" (UID: "3172c7c9-d194-4f02-ac09-9dd878f7df79"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 07:07:46.606100 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.606064 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3172c7c9-d194-4f02-ac09-9dd878f7df79-feast-data" (OuterVolumeSpecName: "feast-data") pod "3172c7c9-d194-4f02-ac09-9dd878f7df79" (UID: "3172c7c9-d194-4f02-ac09-9dd878f7df79"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 07:07:46.608196 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.608166 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-online-tls" (OuterVolumeSpecName: "online-tls") pod "3172c7c9-d194-4f02-ac09-9dd878f7df79" (UID: "3172c7c9-d194-4f02-ac09-9dd878f7df79"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:07:46.610423 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.610397 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "3172c7c9-d194-4f02-ac09-9dd878f7df79" (UID: "3172c7c9-d194-4f02-ac09-9dd878f7df79"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:07:46.610502 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.610425 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "3172c7c9-d194-4f02-ac09-9dd878f7df79" (UID: "3172c7c9-d194-4f02-ac09-9dd878f7df79"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:07:46.610543 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.610499 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3172c7c9-d194-4f02-ac09-9dd878f7df79-kube-api-access-l2gwx" (OuterVolumeSpecName: "kube-api-access-l2gwx") pod "3172c7c9-d194-4f02-ac09-9dd878f7df79" (UID: "3172c7c9-d194-4f02-ac09-9dd878f7df79"). InnerVolumeSpecName "kube-api-access-l2gwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 07:07:46.643547 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.643498 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:07:46.707033 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707003 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-ui-tls\") pod \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " Sep 09 07:07:46.707151 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707044 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-registry-tls\") pod \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " Sep 09 07:07:46.707151 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707090 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-feast-data\") pod \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " Sep 09 07:07:46.707151 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707107 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-online-tls\") pod \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " Sep 09 07:07:46.707151 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707139 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq6sr\" (UniqueName: \"kubernetes.io/projected/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-kube-api-access-jq6sr\") pod \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " Sep 09 07:07:46.707351 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707172 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-offline-tls\") pod \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\" (UID: \"6e842f35-4b92-4a2c-b34d-7bf05c0353d4\") " Sep 09 07:07:46.707416 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707386 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3172c7c9-d194-4f02-ac09-9dd878f7df79-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.707416 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707404 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.707533 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707416 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3172c7c9-d194-4f02-ac09-9dd878f7df79-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.707533 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707428 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.707533 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707441 2563 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3172c7c9-d194-4f02-ac09-9dd878f7df79-ui-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.707533 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707453 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2gwx\" (UniqueName: \"kubernetes.io/projected/3172c7c9-d194-4f02-ac09-9dd878f7df79-kube-api-access-l2gwx\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.707683 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.707646 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-feast-data" (OuterVolumeSpecName: "feast-data") pod "6e842f35-4b92-4a2c-b34d-7bf05c0353d4" (UID: "6e842f35-4b92-4a2c-b34d-7bf05c0353d4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 07:07:46.709393 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.709367 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-kube-api-access-jq6sr" (OuterVolumeSpecName: "kube-api-access-jq6sr") pod "6e842f35-4b92-4a2c-b34d-7bf05c0353d4" (UID: "6e842f35-4b92-4a2c-b34d-7bf05c0353d4"). InnerVolumeSpecName "kube-api-access-jq6sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 07:07:46.709841 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.709817 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-online-tls" (OuterVolumeSpecName: "online-tls") pod "6e842f35-4b92-4a2c-b34d-7bf05c0353d4" (UID: "6e842f35-4b92-4a2c-b34d-7bf05c0353d4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:07:46.709908 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.709836 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6e842f35-4b92-4a2c-b34d-7bf05c0353d4" (UID: "6e842f35-4b92-4a2c-b34d-7bf05c0353d4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:07:46.709908 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.709852 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6e842f35-4b92-4a2c-b34d-7bf05c0353d4" (UID: "6e842f35-4b92-4a2c-b34d-7bf05c0353d4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:07:46.709908 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.709871 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "6e842f35-4b92-4a2c-b34d-7bf05c0353d4" (UID: "6e842f35-4b92-4a2c-b34d-7bf05c0353d4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 07:07:46.808241 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.808215 2563 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-feast-data\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.808241 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.808240 2563 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-online-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.808392 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.808253 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq6sr\" (UniqueName: \"kubernetes.io/projected/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-kube-api-access-jq6sr\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.808392 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.808265 2563 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-offline-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.808392 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.808276 2563 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-ui-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:46.808392 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:46.808287 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e842f35-4b92-4a2c-b34d-7bf05c0353d4-registry-tls\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:07:47.025599 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.025532 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" Sep 09 07:07:47.025599 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.025569 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47" event={"ID":"3172c7c9-d194-4f02-ac09-9dd878f7df79","Type":"ContainerDied","Data":"f307afc063f76c6bea1d7b7cba1a4dc752f3f7cd59f404b5db60ed92c0ef0b70"} Sep 09 07:07:47.026008 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.025629 2563 scope.go:117] "RemoveContainer" containerID="a4dd96f411b8035dcca086efe416ab3a635ede7639501fb14b04d22ad818b4a1" Sep 09 07:07:47.028941 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.028920 2563 generic.go:356] "Generic (PLEG): container finished" podID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerID="ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47" exitCode=137 Sep 09 07:07:47.028941 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.028938 2563 generic.go:356] "Generic (PLEG): container finished" podID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerID="3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669" exitCode=137 Sep 09 07:07:47.029100 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.028955 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerDied","Data":"ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47"} Sep 09 07:07:47.029100 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.029012 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerDied","Data":"3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669"} Sep 09 07:07:47.029100 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.029028 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" event={"ID":"6e842f35-4b92-4a2c-b34d-7bf05c0353d4","Type":"ContainerDied","Data":"af914a0409eda0d37e37de47bd2a2a73c7e6db0780f4c08323351c541f907039"} Sep 09 07:07:47.029100 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.029033 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr" Sep 09 07:07:47.035346 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.035331 2563 scope.go:117] "RemoveContainer" containerID="8221d1d33586be6f9b7057cf097a695556a8e3203bd1c93cc4211d26de7fed1f" Sep 09 07:07:47.043450 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.043426 2563 scope.go:117] "RemoveContainer" containerID="f6463919b4a79930b810301959df87b4453e0f6fd8a73597216c1ca5b312e2a5" Sep 09 07:07:47.050867 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.050836 2563 scope.go:117] "RemoveContainer" containerID="d5ef4ea3b8e3c9266ad2f398040567b5c236103775719fadc4aa07329e2ec218" Sep 09 07:07:47.052853 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.052831 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47"] Sep 09 07:07:47.056649 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.056630 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-28v47"] Sep 09 07:07:47.059863 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.059830 2563 scope.go:117] "RemoveContainer" containerID="2fa60b25f815832ab58467a1845f83a7cca35047366320a56b0e783b81cef454" Sep 09 07:07:47.068133 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.068113 2563 scope.go:117] "RemoveContainer" containerID="c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16" Sep 09 07:07:47.070364 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.070346 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr"] Sep 09 07:07:47.075243 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.075225 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fjhkr"] Sep 09 07:07:47.076412 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.076394 2563 scope.go:117] "RemoveContainer" containerID="ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47" Sep 09 07:07:47.083799 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.083784 2563 scope.go:117] "RemoveContainer" containerID="775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673" Sep 09 07:07:47.091038 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.091019 2563 scope.go:117] "RemoveContainer" containerID="3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669" Sep 09 07:07:47.097818 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.097804 2563 scope.go:117] "RemoveContainer" containerID="031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b" Sep 09 07:07:47.104630 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.104611 2563 scope.go:117] "RemoveContainer" containerID="c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee" Sep 09 07:07:47.111523 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.111508 2563 scope.go:117] "RemoveContainer" containerID="c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16" Sep 09 07:07:47.111789 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:07:47.111772 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16\": container with ID starting with c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16 not found: ID does not exist" containerID="c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16" Sep 09 07:07:47.111854 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.111796 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16"} err="failed to get container status \"c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16\": rpc error: code = NotFound desc = could not find container \"c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16\": container with ID starting with c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16 not found: ID does not exist" Sep 09 07:07:47.111854 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.111814 2563 scope.go:117] "RemoveContainer" containerID="ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47" Sep 09 07:07:47.112123 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:07:47.112094 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47\": container with ID starting with ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47 not found: ID does not exist" containerID="ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47" Sep 09 07:07:47.112180 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.112127 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47"} err="failed to get container status \"ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47\": rpc error: code = NotFound desc = could not find container \"ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47\": container with ID starting with ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47 not found: ID does not exist" Sep 09 07:07:47.112180 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.112151 2563 scope.go:117] "RemoveContainer" containerID="775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673" Sep 09 07:07:47.112393 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:07:47.112378 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673\": container with ID starting with 775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673 not found: ID does not exist" containerID="775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673" Sep 09 07:07:47.112435 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.112396 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673"} err="failed to get container status \"775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673\": rpc error: code = NotFound desc = could not find container \"775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673\": container with ID starting with 775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673 not found: ID does not exist" Sep 09 07:07:47.112435 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.112409 2563 scope.go:117] "RemoveContainer" containerID="3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669" Sep 09 07:07:47.112643 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:07:47.112629 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669\": container with ID starting with 3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669 not found: ID does not exist" containerID="3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669" Sep 09 07:07:47.112694 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.112645 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669"} err="failed to get container status \"3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669\": rpc error: code = NotFound desc = could not find container \"3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669\": container with ID starting with 3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669 not found: ID does not exist" Sep 09 07:07:47.112694 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.112657 2563 scope.go:117] "RemoveContainer" containerID="031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b" Sep 09 07:07:47.112888 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:07:47.112871 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b\": container with ID starting with 031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b not found: ID does not exist" containerID="031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b" Sep 09 07:07:47.112941 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.112893 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b"} err="failed to get container status \"031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b\": rpc error: code = NotFound desc = could not find container \"031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b\": container with ID starting with 031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b not found: ID does not exist" Sep 09 07:07:47.112941 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.112907 2563 scope.go:117] "RemoveContainer" containerID="c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee" Sep 09 07:07:47.113176 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:07:47.113160 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee\": container with ID starting with c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee not found: ID does not exist" containerID="c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee" Sep 09 07:07:47.113213 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113181 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee"} err="failed to get container status \"c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee\": rpc error: code = NotFound desc = could not find container \"c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee\": container with ID starting with c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee not found: ID does not exist" Sep 09 07:07:47.113213 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113194 2563 scope.go:117] "RemoveContainer" containerID="c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16" Sep 09 07:07:47.113434 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113418 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16"} err="failed to get container status \"c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16\": rpc error: code = NotFound desc = could not find container \"c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16\": container with ID starting with c6cd6f994165a688bd763fb6d48f566c0dc3fe9bdc4ac8ebe6be6cc5d2aafd16 not found: ID does not exist" Sep 09 07:07:47.113479 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113434 2563 scope.go:117] "RemoveContainer" containerID="ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47" Sep 09 07:07:47.113611 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113596 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47"} err="failed to get container status \"ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47\": rpc error: code = NotFound desc = could not find container \"ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47\": container with ID starting with ba5eb59552d594e5c1e9ad4606752c3933ef0aa44b561762453138d83604da47 not found: ID does not exist" Sep 09 07:07:47.113649 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113611 2563 scope.go:117] "RemoveContainer" containerID="775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673" Sep 09 07:07:47.113797 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113781 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673"} err="failed to get container status \"775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673\": rpc error: code = NotFound desc = could not find container \"775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673\": container with ID starting with 775127d3dab4ee94eab5dd87d1244806720bf8a8f90539674163bc497065d673 not found: ID does not exist" Sep 09 07:07:47.113844 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113797 2563 scope.go:117] "RemoveContainer" containerID="3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669" Sep 09 07:07:47.113987 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113947 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669"} err="failed to get container status \"3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669\": rpc error: code = NotFound desc = could not find container \"3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669\": container with ID starting with 3841891990d11b92a416fd3a0e80d27c15da27b1243371a4083dfd3051412669 not found: ID does not exist" Sep 09 07:07:47.113987 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.113979 2563 scope.go:117] "RemoveContainer" containerID="031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b" Sep 09 07:07:47.114182 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.114168 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b"} err="failed to get container status \"031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b\": rpc error: code = NotFound desc = could not find container \"031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b\": container with ID starting with 031e45621917cf9ac7ae0e0bdcc1b5ce5a8b6339786dceb844db0cea48b2d72b not found: ID does not exist" Sep 09 07:07:47.114226 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.114182 2563 scope.go:117] "RemoveContainer" containerID="c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee" Sep 09 07:07:47.114369 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:47.114355 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee"} err="failed to get container status \"c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee\": rpc error: code = NotFound desc = could not find container \"c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee\": container with ID starting with c98195cc058d7b50095614bdbfdff8dc1e38e4c37363a639a457065cdfd632ee not found: ID does not exist" Sep 09 07:07:48.147013 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:48.146983 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" path="/var/lib/kubelet/pods/3172c7c9-d194-4f02-ac09-9dd878f7df79/volumes" Sep 09 07:07:48.147519 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:07:48.147505 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" path="/var/lib/kubelet/pods/6e842f35-4b92-4a2c-b34d-7bf05c0353d4/volumes" Sep 09 07:08:05.322831 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.322798 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jhxxv/must-gather-29m6c"] Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323117 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="feast-apply" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323129 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="feast-apply" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323137 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="offline" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323142 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="offline" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323149 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="online" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323154 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="online" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323163 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="ui" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323168 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="ui" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323174 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="ui" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323179 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="ui" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323187 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="feast-init" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323192 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="feast-init" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323204 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="feast-init" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323209 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="feast-init" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323214 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="feast-apply" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323218 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="feast-apply" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323226 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="offline" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323231 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="offline" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323237 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323241 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323247 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="registry" Sep 09 07:08:05.323266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323252 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="registry" Sep 09 07:08:05.324024 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323312 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="ui" Sep 09 07:08:05.324024 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323323 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="registry" Sep 09 07:08:05.324024 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323332 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="online" Sep 09 07:08:05.324024 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323341 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="ui" Sep 09 07:08:05.324024 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323347 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="online" Sep 09 07:08:05.324024 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323353 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3172c7c9-d194-4f02-ac09-9dd878f7df79" containerName="offline" Sep 09 07:08:05.324024 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.323358 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e842f35-4b92-4a2c-b34d-7bf05c0353d4" containerName="offline" Sep 09 07:08:05.326006 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.325990 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:05.328947 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.328921 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhxxv\"/\"kube-root-ca.crt\"" Sep 09 07:08:05.328947 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.328942 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhxxv\"/\"openshift-service-ca.crt\"" Sep 09 07:08:05.329134 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.328942 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jhxxv\"/\"default-dockercfg-dzdkj\"" Sep 09 07:08:05.337044 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.337024 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhxxv/must-gather-29m6c"] Sep 09 07:08:05.342731 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.342711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-must-gather-output\") pod \"must-gather-29m6c\" (UID: \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\") " pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:05.342809 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.342761 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nztk\" (UniqueName: \"kubernetes.io/projected/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-kube-api-access-5nztk\") pod \"must-gather-29m6c\" (UID: \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\") " pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:05.443632 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.443609 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-must-gather-output\") pod \"must-gather-29m6c\" (UID: \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\") " pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:05.443797 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.443677 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nztk\" (UniqueName: \"kubernetes.io/projected/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-kube-api-access-5nztk\") pod \"must-gather-29m6c\" (UID: \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\") " pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:05.443939 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.443917 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-must-gather-output\") pod \"must-gather-29m6c\" (UID: \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\") " pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:05.454497 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.454476 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nztk\" (UniqueName: \"kubernetes.io/projected/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-kube-api-access-5nztk\") pod \"must-gather-29m6c\" (UID: \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\") " pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:05.650625 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.650599 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:05.921235 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.921080 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhxxv/must-gather-29m6c"] Sep 09 07:08:05.924050 ip-10-0-128-45 kubenswrapper[2563]: W0909 07:08:05.924025 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6faf07af_4c5f_4fb5_86b8_be8fbf29d0d9.slice/crio-c9233924241a04618696ac0588898eb23901c18a32e0e464e34f384000354a34 WatchSource:0}: Error finding container c9233924241a04618696ac0588898eb23901c18a32e0e464e34f384000354a34: Status 404 returned error can't find the container with id c9233924241a04618696ac0588898eb23901c18a32e0e464e34f384000354a34 Sep 09 07:08:05.925950 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:05.925931 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 07:08:06.104586 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:06.104549 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhxxv/must-gather-29m6c" event={"ID":"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9","Type":"ContainerStarted","Data":"c9233924241a04618696ac0588898eb23901c18a32e0e464e34f384000354a34"} Sep 09 07:08:12.134236 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:12.134199 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhxxv/must-gather-29m6c" event={"ID":"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9","Type":"ContainerStarted","Data":"8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3"} Sep 09 07:08:13.140604 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:13.140570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhxxv/must-gather-29m6c" event={"ID":"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9","Type":"ContainerStarted","Data":"4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b"} Sep 09 07:08:13.157447 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:13.157398 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jhxxv/must-gather-29m6c" podStartSLOduration=2.110874661 podStartE2EDuration="8.157382021s" totalImagesPullingTime="6.04650736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:08:13.155914346 +0000 UTC m=+2117.480724804" watchObservedRunningTime="2026-09-09 07:08:13.157382021 +0000 UTC m=+2117.482192477" Sep 09 07:08:22.180396 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:22.180362 2563 generic.go:356] "Generic (PLEG): container finished" podID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" containerID="8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3" exitCode=0 Sep 09 07:08:22.180877 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:22.180429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhxxv/must-gather-29m6c" event={"ID":"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9","Type":"ContainerDied","Data":"8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3"} Sep 09 07:08:22.180877 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:22.180711 2563 scope.go:117] "RemoveContainer" containerID="8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3" Sep 09 07:08:22.546130 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:22.546049 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jhxxv_must-gather-29m6c_6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9/gather/0.log" Sep 09 07:08:25.953121 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:25.953084 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f952v_e9bc0891-f9b1-469c-9337-c7832bb1d7ae/global-pull-secret-syncer/0.log" Sep 09 07:08:26.028493 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:26.028468 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cq5mp_084a6df0-80e1-466e-a084-78b251adc62f/konnectivity-agent/0.log" Sep 09 07:08:26.083504 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:26.083476 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-45.ec2.internal_ce4b5beeab7b2f9c4597d30a91deced8/haproxy/0.log" Sep 09 07:08:27.853092 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:27.853053 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jhxxv/must-gather-29m6c"] Sep 09 07:08:27.853488 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:27.853232 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-jhxxv/must-gather-29m6c" podUID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" containerName="copy" containerID="cri-o://4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b" gracePeriod=2 Sep 09 07:08:27.860275 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:27.860245 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jhxxv/must-gather-29m6c"] Sep 09 07:08:27.860733 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:08:27.860698 2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-jhxxv\"/\"must-gather-29m6c\": pods \"must-gather-29m6c\" not found" pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:28.085796 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.085772 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jhxxv_must-gather-29m6c_6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9/copy/0.log" Sep 09 07:08:28.087400 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.087385 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:28.089694 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:08:28.089669 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-29m6c\" is forbidden: User \"system:node:ip-10-0-128-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jhxxv\": no relationship found between node 'ip-10-0-128-45.ec2.internal' and this object" podUID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:28.205666 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.205590 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jhxxv_must-gather-29m6c_6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9/copy/0.log" Sep 09 07:08:28.205997 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.205972 2563 generic.go:356] "Generic (PLEG): container finished" podID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" containerID="4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b" exitCode=143 Sep 09 07:08:28.206098 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.206036 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhxxv/must-gather-29m6c" Sep 09 07:08:28.206158 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.206042 2563 scope.go:117] "RemoveContainer" containerID="4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b" Sep 09 07:08:28.215476 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.215453 2563 scope.go:117] "RemoveContainer" containerID="8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3" Sep 09 07:08:28.227898 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.227881 2563 scope.go:117] "RemoveContainer" containerID="4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b" Sep 09 07:08:28.228180 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:08:28.228164 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b\": container with ID starting with 4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b not found: ID does not exist" containerID="4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b" Sep 09 07:08:28.228235 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.228190 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b"} err="failed to get container status \"4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b\": rpc error: code = NotFound desc = could not find container \"4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b\": container with ID starting with 4062626ecf4bd0bd59b68299ae97d57fb247ab9b7434e465333f73e8d9d5162b not found: ID does not exist" Sep 09 07:08:28.228235 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.228207 2563 scope.go:117] "RemoveContainer" containerID="8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3" Sep 09 07:08:28.228324 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.228206 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nztk\" (UniqueName: \"kubernetes.io/projected/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-kube-api-access-5nztk\") pod \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\" (UID: \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\") " Sep 09 07:08:28.228378 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.228334 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-must-gather-output\") pod \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\" (UID: \"6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9\") " Sep 09 07:08:28.228515 ip-10-0-128-45 kubenswrapper[2563]: E0909 07:08:28.228476 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3\": container with ID starting with 8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3 not found: ID does not exist" containerID="8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3" Sep 09 07:08:28.228515 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.228502 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3"} err="failed to get container status \"8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3\": rpc error: code = NotFound desc = could not find container \"8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3\": container with ID starting with 8bb62e510a736401dc66ae7c6dab2e77cc351e016b2152db0b9388cd4f80e4b3 not found: ID does not exist" Sep 09 07:08:28.228686 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.228666 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" (UID: "6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 07:08:28.230373 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.230354 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-kube-api-access-5nztk" (OuterVolumeSpecName: "kube-api-access-5nztk") pod "6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" (UID: "6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9"). InnerVolumeSpecName "kube-api-access-5nztk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 07:08:28.329641 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.329603 2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-must-gather-output\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:08:28.329641 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:28.329635 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nztk\" (UniqueName: \"kubernetes.io/projected/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9-kube-api-access-5nztk\") on node \"ip-10-0-128-45.ec2.internal\" DevicePath \"\"" Sep 09 07:08:30.147547 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.147514 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" path="/var/lib/kubelet/pods/6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9/volumes" Sep 09 07:08:30.250570 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.250533 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6cgcw_41d8bb2e-ae23-407a-9218-019bfc151181/kube-state-metrics/0.log" Sep 09 07:08:30.272897 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.272866 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6cgcw_41d8bb2e-ae23-407a-9218-019bfc151181/kube-rbac-proxy-main/0.log" Sep 09 07:08:30.294266 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.294243 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6cgcw_41d8bb2e-ae23-407a-9218-019bfc151181/kube-rbac-proxy-self/0.log" Sep 09 07:08:30.323883 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.323857 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-75f55d8478-dh8kl_b89c9acd-a35c-4085-af18-5a7d4b183d45/metrics-server/0.log" Sep 09 07:08:30.348381 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.348357 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-pvt4m_5baef439-4b19-4422-acce-8b0ca7bade33/monitoring-plugin/0.log" Sep 09 07:08:30.470165 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.470094 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h677h_daf1b317-03e2-4071-ae5e-abe256128da5/node-exporter/0.log" Sep 09 07:08:30.497245 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.497225 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h677h_daf1b317-03e2-4071-ae5e-abe256128da5/kube-rbac-proxy/0.log" Sep 09 07:08:30.517832 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.517814 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h677h_daf1b317-03e2-4071-ae5e-abe256128da5/init-textfile/0.log" Sep 09 07:08:30.624104 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.624074 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fb6wr_5c35f04b-688c-4f62-b55f-d757aeb29653/kube-rbac-proxy-main/0.log" Sep 09 07:08:30.646665 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.646646 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fb6wr_5c35f04b-688c-4f62-b55f-d757aeb29653/kube-rbac-proxy-self/0.log" Sep 09 07:08:30.671227 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.671205 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fb6wr_5c35f04b-688c-4f62-b55f-d757aeb29653/openshift-state-metrics/0.log" Sep 09 07:08:30.891718 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.891691 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-k5x9g_f3a84eba-7709-40c9-987a-f1dba4a79cd8/prometheus-operator/0.log" Sep 09 07:08:30.922880 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.922860 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-k5x9g_f3a84eba-7709-40c9-987a-f1dba4a79cd8/kube-rbac-proxy/0.log" Sep 09 07:08:30.950602 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:30.950582 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-8krnx_f8fbc92c-020b-4485-9726-25672578965f/prometheus-operator-admission-webhook/0.log" Sep 09 07:08:33.322479 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.322448 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w"] Sep 09 07:08:33.322992 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.322870 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" containerName="copy" Sep 09 07:08:33.322992 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.322888 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" containerName="copy" Sep 09 07:08:33.322992 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.322909 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" containerName="gather" Sep 09 07:08:33.322992 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.322917 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" containerName="gather" Sep 09 07:08:33.323217 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.323012 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" containerName="gather" Sep 09 07:08:33.323217 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.323028 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6faf07af-4c5f-4fb5-86b8-be8fbf29d0d9" containerName="copy" Sep 09 07:08:33.325164 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.325143 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.330520 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.330498 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvhpm\"/\"openshift-service-ca.crt\"" Sep 09 07:08:33.330636 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.330551 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fvhpm\"/\"default-dockercfg-59lkm\"" Sep 09 07:08:33.331814 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.331603 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvhpm\"/\"kube-root-ca.crt\"" Sep 09 07:08:33.340334 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.340311 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w"] Sep 09 07:08:33.438754 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.438729 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5fdb6b786f-7lws7_0e6a14f8-7d19-4d94-91cf-dd9beb024918/console/0.log" Sep 09 07:08:33.470207 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.470178 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-lib-modules\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.470348 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.470219 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-sys\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.470348 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.470236 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-podres\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.470348 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.470252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6bzb\" (UniqueName: \"kubernetes.io/projected/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-kube-api-access-s6bzb\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.470348 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.470271 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-proc\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.475242 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.475223 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-7nh4z_dc238f64-8c8b-4a1f-a9fa-d3a3c6ab2919/download-server/0.log" Sep 09 07:08:33.571485 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.571455 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-lib-modules\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.571622 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.571502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-sys\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.571622 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.571571 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-sys\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.571622 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.571611 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-podres\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.571747 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.571631 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-lib-modules\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.571747 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.571642 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6bzb\" (UniqueName: \"kubernetes.io/projected/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-kube-api-access-s6bzb\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.571747 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.571666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-proc\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.571747 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.571693 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-podres\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.571747 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.571732 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-proc\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.580480 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.580425 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6bzb\" (UniqueName: \"kubernetes.io/projected/b6ba2c6f-aa54-4e55-ade9-47cf2461a103-kube-api-access-s6bzb\") pod \"perf-node-gather-daemonset-fjr6w\" (UID: \"b6ba2c6f-aa54-4e55-ade9-47cf2461a103\") " pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.635275 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.635244 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:33.902084 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.902055 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w"] Sep 09 07:08:33.905269 ip-10-0-128-45 kubenswrapper[2563]: W0909 07:08:33.905241 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb6ba2c6f_aa54_4e55_ade9_47cf2461a103.slice/crio-64465ab7d28cd69481ee0696d2cf770500def1ef708b7e6731673ca1218af4f5 WatchSource:0}: Error finding container 64465ab7d28cd69481ee0696d2cf770500def1ef708b7e6731673ca1218af4f5: Status 404 returned error can't find the container with id 64465ab7d28cd69481ee0696d2cf770500def1ef708b7e6731673ca1218af4f5 Sep 09 07:08:33.939810 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:33.939789 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-jx9q7_07921b1d-7654-4c49-81cb-d0716848377b/volume-data-source-validator/0.log" Sep 09 07:08:34.231457 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:34.231377 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" event={"ID":"b6ba2c6f-aa54-4e55-ade9-47cf2461a103","Type":"ContainerStarted","Data":"c35bd5d04de939db2539bd7625960b041c741d535342e0d74537f3c4123b4a1f"} Sep 09 07:08:34.231457 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:34.231425 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" event={"ID":"b6ba2c6f-aa54-4e55-ade9-47cf2461a103","Type":"ContainerStarted","Data":"64465ab7d28cd69481ee0696d2cf770500def1ef708b7e6731673ca1218af4f5"} Sep 09 07:08:34.231457 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:34.231456 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:34.249981 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:34.249911 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" podStartSLOduration=1.249897211 podStartE2EDuration="1.249897211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 07:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:08:34.248863269 +0000 UTC m=+2138.573673735" watchObservedRunningTime="2026-09-09 07:08:34.249897211 +0000 UTC m=+2138.574707667" Sep 09 07:08:34.804413 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:34.804383 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d8257_784894d0-7e1e-4611-9e85-c737ddbe3cad/dns/0.log" Sep 09 07:08:34.826369 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:34.826338 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d8257_784894d0-7e1e-4611-9e85-c737ddbe3cad/kube-rbac-proxy/0.log" Sep 09 07:08:34.927741 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:34.927715 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sv5qm_3527edfe-7c0e-4ab9-8357-d7e4e1fee8ec/dns-node-resolver/0.log" Sep 09 07:08:35.381218 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:35.381179 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-plfqc_eb076d84-0bb2-474a-a6d7-c83e1c98904c/node-ca/0.log" Sep 09 07:08:36.448083 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:36.448054 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8t8b_02718a56-6748-4bb7-9cef-01d5fe02132a/serve-healthcheck-canary/0.log" Sep 09 07:08:36.850002 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:36.849956 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bl97r_6b6445dd-fad0-49e5-aeaa-98bd2849a12c/kube-rbac-proxy/0.log" Sep 09 07:08:36.875367 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:36.875347 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bl97r_6b6445dd-fad0-49e5-aeaa-98bd2849a12c/exporter/0.log" Sep 09 07:08:36.901351 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:36.901324 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bl97r_6b6445dd-fad0-49e5-aeaa-98bd2849a12c/extractor/0.log" Sep 09 07:08:40.245532 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:40.245500 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fvhpm/perf-node-gather-daemonset-fjr6w" Sep 09 07:08:41.246127 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:41.246101 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-bz9hs_8d403b3f-a880-428c-bd1b-e39c557c9e97/migrator/0.log" Sep 09 07:08:41.269888 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:41.269859 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-bz9hs_8d403b3f-a880-428c-bd1b-e39c557c9e97/graceful-termination/0.log" Sep 09 07:08:41.577931 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:41.577855 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-f4h66_ace615ac-3fae-4d50-9e0b-e5b3d149f51f/kube-storage-version-migrator-operator/1.log" Sep 09 07:08:41.578604 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:41.578583 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-f4h66_ace615ac-3fae-4d50-9e0b-e5b3d149f51f/kube-storage-version-migrator-operator/0.log" Sep 09 07:08:42.767080 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:42.767005 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgq4b_c588c0a2-8028-45ef-bf1a-01f4c8a167b1/kube-multus-additional-cni-plugins/0.log" Sep 09 07:08:42.792426 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:42.792406 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgq4b_c588c0a2-8028-45ef-bf1a-01f4c8a167b1/egress-router-binary-copy/0.log" Sep 09 07:08:42.815147 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:42.815125 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgq4b_c588c0a2-8028-45ef-bf1a-01f4c8a167b1/cni-plugins/0.log" Sep 09 07:08:42.840683 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:42.840657 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgq4b_c588c0a2-8028-45ef-bf1a-01f4c8a167b1/bond-cni-plugin/0.log" Sep 09 07:08:42.864669 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:42.864629 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgq4b_c588c0a2-8028-45ef-bf1a-01f4c8a167b1/routeoverride-cni/0.log" Sep 09 07:08:42.888107 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:42.888086 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgq4b_c588c0a2-8028-45ef-bf1a-01f4c8a167b1/whereabouts-cni-bincopy/0.log" Sep 09 07:08:42.910384 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:42.910359 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgq4b_c588c0a2-8028-45ef-bf1a-01f4c8a167b1/whereabouts-cni/0.log" Sep 09 07:08:43.152513 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:43.152482 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-th78r_fac2dd48-8f12-4fe4-ae27-980d0d67772a/kube-multus/0.log" Sep 09 07:08:43.300139 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:43.300063 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p8hkq_00c134a2-09bb-4d7a-abaa-8a81a37d2459/network-metrics-daemon/0.log" Sep 09 07:08:43.320280 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:43.320259 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p8hkq_00c134a2-09bb-4d7a-abaa-8a81a37d2459/kube-rbac-proxy/0.log" Sep 09 07:08:44.443242 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:44.443212 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vr8f9_a249fd1c-7ac3-43b4-b802-4e8cb39b1a89/ovn-controller/0.log" Sep 09 07:08:44.463080 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:44.463054 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vr8f9_a249fd1c-7ac3-43b4-b802-4e8cb39b1a89/ovn-acl-logging/0.log" Sep 09 07:08:44.491750 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:44.491711 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vr8f9_a249fd1c-7ac3-43b4-b802-4e8cb39b1a89/kube-rbac-proxy-node/0.log" Sep 09 07:08:44.512185 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:44.512153 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vr8f9_a249fd1c-7ac3-43b4-b802-4e8cb39b1a89/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 07:08:44.530740 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:44.530712 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vr8f9_a249fd1c-7ac3-43b4-b802-4e8cb39b1a89/northd/0.log" Sep 09 07:08:44.553541 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:44.553510 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vr8f9_a249fd1c-7ac3-43b4-b802-4e8cb39b1a89/nbdb/0.log" Sep 09 07:08:44.572354 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:44.572332 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vr8f9_a249fd1c-7ac3-43b4-b802-4e8cb39b1a89/sbdb/0.log" Sep 09 07:08:44.610555 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:44.610530 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vr8f9_a249fd1c-7ac3-43b4-b802-4e8cb39b1a89/ovnkube-controller/0.log" Sep 09 07:08:45.555755 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:45.555720 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-bs7cf_f45f5765-d893-4325-a7d7-83dd1ad5a627/check-endpoints/0.log" Sep 09 07:08:45.604606 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:45.604571 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mhgvf_2b9d7dfe-79ff-4abf-a090-7df31ea02020/network-check-target-container/0.log" Sep 09 07:08:46.462942 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:46.462906 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gpwd6_cd36bba0-b482-4d51-9ab7-8b045c8f6006/iptables-alerter/0.log" Sep 09 07:08:47.154508 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:47.154481 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-fxtp9_dd7ad407-6b70-4e58-aeb1-538a531a2067/tuned/0.log" Sep 09 07:08:48.911485 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:48.911404 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-hlb8s_77cf1306-5823-4ed6-826c-2fad59e20080/cluster-samples-operator/0.log" Sep 09 07:08:48.929648 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:48.929627 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-hlb8s_77cf1306-5823-4ed6-826c-2fad59e20080/cluster-samples-operator-watch/0.log" Sep 09 07:08:50.014331 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:50.014302 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-knzv9_af365614-5695-450a-a777-f25e24f4d564/service-ca-operator/1.log" Sep 09 07:08:50.015273 ip-10-0-128-45 kubenswrapper[2563]: I0909 07:08:50.015252 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-knzv9_af365614-5695-450a-a777-f25e24f4d564/service-ca-operator/0.log"